Spectral Kurtosis-Based RFI Mitigation for CHIME
Abstract
We present the implementation of a spectral kurtosis-based Radio-Frequency Interference detection system on the CHIME instrument and its reduced-scale pathfinder. Our implementation extends single-receiver formulations to the case of a compact array, combining samples from multiple receivers to improve the confidence with which RFI is detected. Through comparison between on-sky data and simulations, we show that the statistical properties of the canonical spectral kurtosis estimator are functionally unchanged by cross-array integration. Moreover, by comparison of simultaneous data from CHIME and the Pathfinder, we evaluate our implementation’s capacity for interference discrimination for compact arrays of various sizes. We conclude that a spectral kurtosis-based implementation provides a scalable, high cadence RFI discriminator for compact multi-receiver arrays.
1. Introduction
The presence of non-astronomical signals in the electromagnetic spectrum (“Radio-Frequency Interference” or RFI) poses a significant hazard to successful radio observations. Sources of such interference include lightning and electrical discharges, inadvertent emissions from terrestrial electronics, radio-frequency telecommunication signals, and communications with satellites and aircraft (Galt, 1991; Offringa et al., 2013; Sokolowski et al., 2016).
The Canadian Hydrogen Intensity Mapping Experiment (CHIME(a)) is a newly-constructed radio interferometer with 1024 dual-polarization receivers continuously observing a 400–800MHz band. A fully independent reduced-scale prototype, the CHIME Pathfinder (henceforth, “the Pathfinder”) possesses 128 dual-polarization receivers and an identical observing band (Bandura et al., 2014; Newburgh et al., 2014). CHIME and the Pathfinder each follow an “FX” correlator architecture, with an FPGA-based Fourier-transform stage (Bandura et al., 2016) followed by a GPU-based outer-product “X-engine” (Klages et al., 2015; Recnik et al., 2015; Denman et al., 2015).
The considerable capabilities of the CHIME correlator present an opportunity for powerful, on-the-fly RFI mitigation. CHIME employs a “software” correlator X-engine, and its easily-reconfigurable, general-purpose hardware permits the introduction of additional data processing with minimal development effort. This includes extending the real-time processing system to include the detection and excision of RFI during the correlation process. The architecture of the CHIME X-engine is such that each processing “node” contains data from only a few frequencies, but over the entire array; this constrains the selection of RFI-excision algorithms which may be applied in the GPUs. Our choice of a statistical excision algorithm, based on spectral kurtosis, was informed by both robust performance and modest computational cost. Additionally, as detailed below, the compact layout of CHIME permits multi-receiver RFI detection, lowering the effective threshold for RFI detections while retaining high time resolution.
We summarize the theoretical foundations of the spectral kurtosis estimator in Sec. 2.1 and present a multi-receiver formulation of the estimator in Sec. 2.2. The details of its implementation on CHIME and the Pathfinder are included in Sec. 3.1 and the results obtained follow in Sec. 3.2.
2. The Spectral Kurtosis Estimator
Tests of the normalized higher moments of a distribution provide a powerful metric for the presence of RFI. Given the expected Gaussianity of the electric fields produced by natural sources, and the extreme non-Gaussianity of observed RFI sources, deviations in the statistical properties of radio measurements signal the presence of substantial artificial contamination.
The use of a kurtosis-based estimator as an indicator of significant non-Gaussianity, and therefore, contamination, has a long history: Dwyer (1983, 1984); Servière (1998); Vrabie et al. (2003) and Nita et al. (2007) explore a spectral-domain kurtosis measure, with further development and radio-astronomical application in Nita & Gary (2010a,b); Gary et al. (2010) and Nita (2016). Spectral-domain kurtosis measurement provides a robust and overall-scaling-invariant probe of the data distribution, with substantial potential for discrimination.
2.1. Spectral kurtosis
For a set of independent complex values , representing the post-Fourier-transform timestream of a single frequency channel as received by the X-engine, we construct an unbiased spectral kurtosis estimator after Nita & Gary (2010a)
For drawn from a circularly-symmetric complex Gaussian, the will be -distributed with two degrees of freedom. Hence, as shown by Nita & Gary (2010a), to first-order the expected mean, variance, skewness, and kurtosis of are
Notably, the expected mean is invariant and the variance, skewness and kurtosis depend solely on .
2.2. Spectral kurtosis in a compact array context
For an interferometer in which the receivers are separated by a maximum distance , and in which the data from each receiver is used to compute a spectral kurtosis estimate over a period , any transient RFI signal will appear “simultaneous” across the array in the case that . This is true of CHIME’s compact configuration: the receivers are separated by at most m, so any RFI signal will arrive at all the receivers within s, far shorter than the timescales at which spectral kurtosis estimates are produced in our implementation. For an array which is compact relative to the distance to sources of RFI, receivers will see the interference at similar levels, permitting their combination to enhance RFI detection.
The discrimination potential of a spectral kurtosis-based estimator is limited by its variance, which depends solely on the total number of samples present in the input data, . Therefore, increasing corresponds to increasing the discrimination potential of the estimator. For a single receiver with a fixed sampling rate, improving the discrimination potential of the estimator necessarily reduces the cadence at which estimates are produced. Alternatively, combining measurements from multiple receivers can increase the size of input dataset (reducing the inherent variance in the estimator) without affecting the cadence at which kurtosis measurements are generated. If the receivers provide uncorrelated samples (see Appendix A for limitations on this) combining data from multiple receivers will increase the effective integration length, reducing the inherent variance in the estimator when calculated at a given cadence.
2.2.1. Multi-receiver formulation of the spectral kurtosis estimator
For a compact array of -independent receivers, in which spectral kurtosis estimates are desired every time-samples, one possible method of combining the information from the different antennas would be to simply average the individual spectral kurtosis estimates after they are produced. We present here, an alternative formulation which produces mathematically equivalent results by combining the intermediate cumulants from different receivers into a single overall estimator.
We begin by considering the spectral kurtosis estimate produced by the th receiver,
After normalizing each receiver’s signal by its mean power , we sum across the array to produce the normalized cumulants and
By substituting the definition of from Eq. (8) into Eq. (11), we may re-express as
We may then construct a multi-receiver spectral kurtosis estimator analogous to that in Eq. (2), using the normalized cumulants and and an effective number of samples as follows:
Equation (13) illustrates that computing a spectral kurtosis estimate in this manner, by directly combining samples across the array, is equivalent to averaging spectral kurtosis estimates generated from the separate receivers. It should be noted that this is only the case if the individual receivers’ data is correctly normalized. The constant factor in front of Eq. (13) motivates a convenient re-scaling of the multi-receiver spectral kurtosis estimator:
The statistical properties of this estimator may be directly recovered from those of the constituent ; particularly, the expected mean and variance are
2.2.2. Statistical biases
The statistical properties described in Secs. 2.1 and 2.2.1 are precisely correct in the case of continuous sampling of the underlying Gaussian distributions; this requirement is not formally met when implementing the estimator in digital hardware. In correlator systems which use discretized representations of the data, the saturation of the representable range by extreme values and the quantization effects of numerical representations may affect the statistical properties of ; we discuss this effect in Appendix B.
2.2.3. Computational cost of the simplified multi-receiver algorithm
The multi-receiver formulation described above has reduced computational requirements relative to the corresponding individual computations. The minimum number of operations required to calculate a spectral kurtosis estimate for time-samples and antennas with and without the combination of receivers are as follows:
Both implementations require several core computations:
Step | Formula | Operations |
---|---|---|
Calculating power | ||
Summing power in time | ||
Summing square power in time | ( | |
Sub-total |
The additional computations required to produce spectral kurtosis estimates in both cases are as follows:
Independent Estimates | ||
---|---|---|
Step | Formula | Operations |
Previous sub-total | ||
Calculating estimates | ||
Calculating average | ||
Total |
Multi-Receiver Formulation | ||
Step | Formula | Operations |
Previous sub-total | ||
Finding mean power | ||
Applying normalization to square power | ||
Summing of square power across receivers | ||
Calculating multi-receiver estimate | 7 | |
Total |
Although in both cases the leading term is of the form , the alternative formulation offers an reduction in the number of operations required. For any value of greater than one, the simplified multi-receiver algorithm requires fewer operations to compute at an identical cadence.
3. Application to CHIME
3.1. Implementation
The CHIME X-engine (Denman et al., in preparation) consists of 256 correlator nodes, each hosting two dual-chip AMD S9300x2 GPUs. Each of the four GPU chips processes time-ordered data for all of CHIME’s 2048 receivers over one kHz frequency channel, and is therefore able to compute estimates for a single sub-band. The Pathfinder’s X-engine has a similar structure; its 16 compute nodes each receive 64 frequency channels from all 256 receivers, and subsequently utilize the same GPU kernels and acquisition software as CHIME to compute estimates.
Two GPU processing kernels were written in Open Computing Language (OpenCL(b)) to compute real-time estimates at a variety of integration lengths. The kernels were then compiled into Heterogeneous System Architecture Code Objects (HSACOs(c)) which are executed by CHIME’s real-time data processing software, kotekan (Renard et al., in preparation). The first kernel computes, accumulates, and normalizes the power and square-power estimates from each independent receiver. The second kernel sums the output from the previous kernel across the array before computing a single estimate for the current time interval.
This model allows for significant flexibility in the duration of time-integration and total number of receivers in the array, and is highly efficient. In the current implementation, full-array estimates are produced for each of CHIME’s 1024 frequency channels after accumulations of 256 time-samples (a cadence of ms). computations currently require of the available computational power of the CHIME GPU X-engine, and may therefore, run in parallel with the primary correlation systems with minimal impact.
3.2. Results
In a June 2018 engineering run, one hour of simultaneous estimates from both CHIME and the Pathfinder were recorded, with the values of output at a cadence of ms. Out of a total 1024 frequency channels across the 400–800MHz band, 892 and 272 were recorded to completion on CHIME and the Pathfinder, respectively. This data provides a point of comparison between observations and the simulations described in Appendix B as well as verifying consistency between the two instruments.
We define “detection significance” as the difference between the measured value of for a given set of time-samples and its simulated expectation, expressed in units of the standard deviation of when applied to RFI-free data. For example, a threshold of “ standard deviations” in detection significance would correspond to removing data outside of the range . The adoption of a symmetric interval in detection significance as the metric by which RFI is indicated implicitly relies on the large- nature of this implementation; thresholds based on the predicted false-alarm probability offer an alternative for smaller- cases in which the skewness of the is significant (see Nita et al. (2016) for an example). Figure 1 depicts a short segment (ms) of “detection significance” values, computed at a cadence of ms, for all available frequency channels. Both fixed-frequency broadcasting signals and a broadband interference pulse (ms in duration) are detected in the segment.

Fig. 1. A ms segment of detection confidence values from 892 out of 1024 of CHIME’s frequency channels. The broadband pulse (at ms) illustrates the estimator’s ability to detect both fixed-frequency and broadband interference; the detection of such pulses highlights the importance of high cadence RFI detection.
3.2.1. Comparison with simulated data
Data from both CHIME and the Pathfinder indicate that frequencies which are believed to be relatively free of RFI adhere closely to simulations; Fig. 2 compares the simulated, RFI-free distribution to CHIME observations of frequency channels with different levels of persistent RFI contamination. Frequencies with low degrees of RFI contamination reproduce the expected distribution while RFI-saturated channels do not. Figure 3 compares the probability distributions of simulated and observed RFI detection significance values (as defined above) in both the CHIME and Pathfinder datasets. Applying a threshold to this value allows the excision of significantly contaminated time-intervals while also setting a well-constrained false-positive rate for non-contaminated data. When applying a 5-standard-deviation detection threshold, we find that our implementation detects RFI in 16–18% of CHIME’s band over the course of a day.

Fig. 2. A comparison of the observed distribution of an RFI-free simulation to that of both a highly-contaminated “RFI-high” frequency channel and a relatively clean “RFI-low” channel. The latter matches the simulated distribution’s statistics quite closely, while the former has completely inconsistent behavior. Both the simulated and observed estimates were computed after accumulations of 256 time samples. The simulated and “RFI-low” distributions have means which are slightly offset from unity as a result of digitization effects, as further discussed in Appendix B.

Fig. 3. The distribution of detection significance in sample data from both CHIME and the Pathfinder, as compared to a prediction from simulated RFI-free data. The excesses at high significance indicate the presence of RFI in both instruments’ data, with the rightward shift in the CHIME data illustrating the improved interference-to-noise ratio obtained by the larger number of receivers.
3.2.2. Inter-instrument comparison
The simultaneous detection of transient events in both CHIME and the Pathfinder supports the interpretation of changes in as measuring exogenous signals. Figure 4 presents a comparison of simultaneous estimates from both instruments across all available frequencies; the two are strongly correlated, with a significance ratio consistent with the sensitivity difference predicted by simulations of the two instruments. We conclude that, for a given frequency and time interval, the two instruments provide consistent measurements of the presence of RFI. Moreover, through examination of RFI transient events, it becomes evident that both instruments are observing nearly identical surroundings.

Fig. 4. The simultaneously-measured RFI detection significance in synchronous data from both CHIME and the Pathfinder, within the overlapping 216 frequency channels. The diagonal line indicates the interference-to-noise ratio increase expected from their relative number of receivers.
4. Conclusion
We have shown that the excision potential of the estimator may be improved through the combination of signals from multiple receivers within a compact array. Our implementation offers a robust, computationally-efficient method for real-time RFI detection and removal for compact arrays. Data from both CHIME and the Pathfinder agree closely with numerical simulations, confirming that the effects of quantization on the estimator in our correlator system are entirely predictable. Particularly, we find that these effects do not substantially alter the integration-length-dependence of the estimator’s statistical moments. We conclude that the multi-receiver formulation of the spectral kurtosis estimator provides high-time-resolution, low-computational-cost RFI excision whose sensitivity improves predictably with the size of the compact array.
Acknowledgments
We are very grateful for the warm reception and skillful help we have received from the staff of the Dominion Radio Astrophysical Observatory, which is operated by the National Research Council of Canada. We acknowledge support and funding from the Natural Sciences and Engineering Research Council of Canada, the Southern Ontario Smart Computing Innovation Platform, the Ontario Centres for Excellence, and the National Research Council of Canada. We further would like to thank the CHIME collaboration for their support and Thoth Technology for the use of the Algonquin Radio Observatory during prototyping stages. We would finally like to thank the reviewer for their helpful and insightful comments.
Appendices
Notes
Appendix A. Common-Mode Contributions to SK̃
The statistical properties of the spectral kurtosis estimator described in Sec. 2.1, depend on the number of samples which are considered. The multi-receiver formulation , described in Sec. 2.2, combines multiple receivers’ samples to reduce the inherent variance in the estimators; however, this relies on the independence of each receiver’s signal. In the presence of sources which contribute substantially to the antenna temperature, the various receivers may be dominated by a common-mode signal, introducing a correlation and reducing the effective number of independent samples used in computing the estimator.
Figure A.1 shows the results of a numerical simulation in which a number of independent Gaussian datasets, representing the noise-dominated receiver timestreams, are augmented by a common-mode Gaussian signal representing the “source”. As the common-mode signal’s amplitude is increased, the variance in increases, eventually reaching the value which would be predicted for a single-receiver timestream with identical properties. Notably, larger- arrays see this effect at lower levels of common-mode signal, placing a sensitivity-dependent upper limit on the size of array for which the multi-receiver formalism will show practical benefits.

Fig. A.1. Changes in the variance of the multi-receiver estimator in the presence of a common-mode signal. At left, each point displays the variance in for 1000 datasets, each with time-samples and a number of receivers ranging from 128 to 4096. Dashed lines at top and bottom show the predicted values of for and each of the simulated values of . At right, the variance is re-expressed as the effective number of receivers , which smoothly declines from the full complement to an effectively-single-receiver case when the common-mode signal dominates the system.
In the case of CHIME, the instantaneous correlation between receivers is rarely above a few percent — the exception occurs at solar transit, where correlations of order unity are observed at some frequencies. As data is generally of low quality during solar transit, and is discarded, this effect is considered negligible for our purposes.
Appendix B. Digitization Effects on SK̃
The quantization of data within a telescope’s digital processing systems may have significant effects on the properties of spectral kurtosis estimators as applied to the resultant data. Nita et al. (2016) examine this effect and address it by constructing a Gamma distribution with empirically-derived shape and scale parameters, and then using the properties of this function to set the appropriate shape factor for their generalized spectral kurtosis estimator. In our implementation, given the large number of samples accumulated and the well-constrained properties of data within the CHIME digital signal processing system, we have opted to generate excision thresholds based directly on numerical simulations of the CHIME correlator’s data characteristics.
To simulate the digitization-related truncation and rounding effects present in the CHIME correlator, a set of sample data was generated which mimicked the properties of CHIME’s intermediate data products -bit complex Gaussian data (bounded between 7 and 7) with an RMS of least-significant bits in each component. Following Sec. 2.2.1, estimates were computed for a variety of integration lengths and array sizes. The statistics of those estimates were averaged over multiple iterations and compared to analogous non-digitized (floating-point) simulations, the results of which are presented in Fig. B.1.

Fig. B.1. Differences between the statistical moments of “digitized” (heavily discretized) and non-digitized estimates for a variety of effective array sizes. For large numbers of inputs, the mean (left) converges to a value which is offset from the expected result by a constant, while the variance (right) converges to the same value in both digitized and non-digitized simulations.
In the case of CHIME-like discretization, digitization effects on the variance (and higher moments) of the estimator vanish for large numbers of inputs. The mean value of the estimator, however, converges to a value slightly offset from unity. This offset is added directly to the value of the RFI excision threshold; as the higher moments are unchanged for the large number of inputs considered, this restores behavior identical to the threshold-determination methods described in Sec. 3.2.
References
- 2014] “Ground-based and Airborne Telescopes V,” Proc. SPIE 9145, 914522. Google Scholar [
- 2016] J. Astronom. Instrument. 5, 1641005. Link, ADS, Google Scholar [
- Denman, N. et al. [in prep]. Google Scholar
- 2015] “A GPU-based correlator X-Engine implemented on the CHIME Pathfinder,” in Proc. IEEE 26th Int. Conf. on Application-Specific Systems, pp. 35–40. Google Scholar [
- 1983] “Detection of non-Gaussian signals by frequency domain kurtosis estimation,” in IEEE Int. Conf. Acoustics, Speech, and Signal Processing, Vol. 8, pp. 607–610. Google Scholar [
- 1984] IEEE J. Oceanic Eng. OE-9, 85. Crossref, ADS, Google Scholar [
- 1991] “Interference with astronomical observations of OH masers from the Soviet Union's GLONASS Satellites,” in Light Pollution, Radio Interference, and Space Debris, Vol. 17, p. 213. ADS, Google Scholar [
- 2010] PASP 122, 560. Crossref, ADS, Google Scholar [
- 2015] “Architectures and Processors,” in Proc. IEEE 26th Int. Conf. Application-Specific Systems, pp. 164–165. Google Scholar [
- 2014] “Ground-based and Airborne Telescopes V,” Proc. SPIE 9145, 91454V. Crossref, Google Scholar [
- 2016] MNRAS 458, 2530. Crossref, ADS, Google Scholar [
- 2010a] PASP 122, 595. Crossref, ADS, Google Scholar [
- 2010b] MNRAS 406, L60. Crossref, ADS, Google Scholar [
- 2016] “Spectral kurtosis statistics of quantized signals,” in 2016 Radio Frequency Interference (RFI), pp. 75–80. Google Scholar [
- 2016] J. Astronom. Instrument. 5, 1641009. Link, ADS, Google Scholar [
- 2007] PASP 119, 805. Crossref, ADS, Google Scholar [
- 2013] MNRAS 435, 584. Crossref, ADS, Google Scholar [
- 2015] “An efficient real-time data pipeline for the CHIME Pathfinder Radio Telescope X-Engine,” in Proc. IEEE 26th Int. Conf. Application-Specific Systems, pp. 57–61. Google Scholar [
- Renard, A. et al. [in prep]. Google Scholar
- 1998] “Feasibility of source separation in frequency domain,” in IEEE Int. Conf. Acoustics, Speech, and Signal Processing, Vol. 4, pp. 2085–2088. Google Scholar [
- Sokolowski, M., Wayth, R. B. & Lewis, M. [2016] arXiv:1610.04696. Google Scholar
- 2003] “Spectral kurtosis: From definition to application,” in 2003 IEEE-EU-RASIP Workshop on Nonlinear Signal and Image Processing. Google Scholar [