Launch

GENERIC — Custom Data / Other Surveys

At its very core, FIREFLY is a Python package that can be run on any input spectrum, provided the minimal required inputs are given. FIREFLY provides a base-level initialisation script, firefly.py, that runs the fitting procedure from the most basic setup and initialisation format. read_firefly.py is also provided to read and plot the output file in its shortest and simplest form.

These generic scripts have been designed to be adapted and redeveloped to read in different data formats you may have, and run FIREFLY on variety of unique data sources. When adapting these base-level FIREFLY scripts to new data, it is recommended to fit only a single galaxy spectrum first. This will help to verify that the input data is being read correctly and the outputs are sensible, before scaling up to larger samples or parallel runs on High Performance Computing (HPC) systems.

Imports — Packages required to launch FIREFLY

FIREFLY is intentionally lightweight and depends mainly on numpy, astropy, and its own internal modules firefly_setup and firefly_models.

# -----------------------------
# firefly.py (core)
# -----------------------------
import sys, os
import time
import numpy as np

# Astropy core I/O + cosmology
from astropy.io import fits
import astropy.cosmology as co

# FIREFLY internals
import firefly_setup as fs
import firefly_models as fm

# -----------------------------
# read_firefly.py (viewer)
# -----------------------------
import sys
import numpy as np
from astropy.io import fits
import matplotlib.pyplot as plt

The key imports that FIREFLY uses are:

• firefly_setup (imported as fs) - Creates the “spec” container that FIREFLY fits. The module handles; filtering, masking, Milky Way dust correction, instrumental resolutions and all input validation
• firefly_models (imported as fm) - Loads the model library and handles model interpolation, dust grid generation, chi-squared minimisation, PDF sampling for age, metallicity, mass, and writing output tables.
• numpy - FIREFLY spectral data structures are simple numpy.ndarray objects. Used for wavelength grids, flux arrays, loading data and numerical operations.
• astropy.cosmology - FIREFLY embeds the Planck15 cosmology by default and is only required if you want cosmology-anchored stellar population ages.
• astropy.io.fits - Critical for the reading or writing of any input/output *.fits file spectral data. Rewuired for the handling of any multi-extension FITS tables (HDUList, BinTableHDU)
• matplotlib.pyplot - Used only in read_firefly.py for basic visualisation of spectra + model fits.
• sys, os, time - Standard Python utilities for paths, script timers, and environment-handling.

Additional imports needed when adapting and scaling-up FIREFLY

These are not required by FIREFLY itself, but may become essential when preparing new types of input data or creating parallel runners:

• argparse - Highly recommended for creating flexible and robust wrapper scripts around FIREFLY
• multiprocessing, concurrent.futures, subprocess - Essential for implementing custom parallelisation strategies and managing multiple processes for distributed computing tasks.
• fcntl - Provides file locking mechanisms, useful for managing concurrent file access in parallel processing.
• pandas - If your results are stored in CSV catalogs or you need fast table joins.

Minimal inputs to FIREFLY

• Wavelength array (Å) (wavelength)
• Flux array (flux) - default units: flux_units = 1e-17
• Error or inverse-variance array (error)
• Redshift (redshift)
• RA, Dec (ra, dec)
• velocity dispersion (vdisp)
• Instrumental resolution (r_instrument)

Initialising FIREFLY

Loading a spectrum (example from firefly.py):

input_file = os.path.join(repo_root, "Data", "example_data", "SDSS_ex", "spec-0266-51602-0001.dat")
data = np.loadtxt(input_file, unpack=True)

wavelength = data[0,:]
flux      = data[1,:]
error     = data[2,:]
restframe_wavelength = wavelength/(1+redshift)

redshift  = 0.021275453
ra= 145.89219 ; dec= 0.059372
vdisp     = 135.89957

# simple constant resolution example (also in firefly.py)
r_instrument = np.full(len(wavelength), 2000.0)

Opening the spectrum inside FIREFLY:

# Define input object to pass data on to firefly modules and initiate run
spec=fs.firefly_setup(input_file,milky_way_reddening=milky_way_reddening, \
                                  N_angstrom_masked=N_angstrom_masked,\
                                  hpf_mode=hpf_mode,data_wave_medium = data_wave_medium)

# Open single spectrum from arrays
spec.openSingleSpectrum(wavelength, flux, error, redshift, ra, dec, vdisp, emlines, r_instrument)

Preparing and running the model fit (from firefly.py):

# Prepare model templates
model = fm.StellarPopulationModel(spec, output_file, cosmo, models = model_key, \
                                          model_libs = model_lib, imfs = imfs, \
                                          age_limits = [age_min,age_max], downgrade_models = True, \
                                          data_wave_medium = data_wave_medium, fit_wave_medium=fit_wave_medium,Z_limits = Z_limits, \
                                          suffix=suffix, use_downgraded_models = False, \
                                          dust_law=dust_law, max_ebv=max_ebv, num_dust_vals=num_dust_vals, \
                                          dust_smoothing_length=dust_smoothing_length,max_iterations=max_iterations, \
                                          pdf_sampling=pdf_sampling, flux_units=flux_units)

# Initiate fit
model.fit_models_to_data()

Writing the output FITS file:

prihdr = fm.pyfits.Header()
prihdr['FILE'] = os.path.basename(output_file)
prihdu = fm.pyfits.PrimaryHDU(header=prihdr)
tables = [prihdu]

tables.append(model.tbhdu)
complete_hdus = fm.pyfits.HDUList(tables)
complete_hdus.writeto(output_file)

Reading + plotting FIREFLY results (read_firefly.py):

# Read in FIREFLY output file (alter as needed using sys.argv or hard-coded paths)
hdul = fits.open(sys.argv[1])
# or
hdul = fits.open("spFly-spec-0266-51602-0001.fits")

data = hdul[1].data

wave  = data['wavelength']
flux  = data['original_data']
model = data['firefly_model']

# Print out some of the header information
hdul.close()
hdul.info()

print('age: '+str(np.around(10**hdul[1].header['age_lightW'],decimals=2))+' Gyr')
print('[Z/H]: '+str(np.around(hdul[1].header['metallicity_lightW'],decimals=2))+' dex')
print('log M/Msun: '+str(np.around(hdul[1].header['stellar_mass'],decimals=2)))
print('E(B-V): '+str(np.around(hdul[1].header['EBV'],decimals=2))+' mag')

import matplotlib.pyplot as plt
plt.plot(wave, flux)
plt.plot(wave, model)
plt.show()

Plotting the star formation history and Z/H distribution

# Plot the star formation history
fig1=plt.figure()
plt.xlim(0,15)
plt.xlabel('lookback time (Gyr)')
plt.ylabel('frequency')
#plt.bar(10**(csp_age),csp_light,width=1,align='center',edgecolor='k',linewidth=2)
plt.bar(10**(csp_age),csp_light,width=1,align='center',alpha=0.5)
plt.scatter(10**(csp_age),csp_light)
plt.show()

# Plot the metallicity distribution
fig2=plt.figure()
plt.xlim(-2,0.5)
plt.xlabel('[Z/H] (dex)')
plt.ylabel('frequency')
#plt.bar(10**(csp_age),csp_light,width=1,align='center',edgecolor='k',linewidth=2)
plt.bar(csp_Z,csp_light,width=0.1,align='center',alpha=0.5)
plt.scatter(csp_Z,csp_light)
plt.show()

Implementing command-line arguments with argparse

Instead of editing hardcoded variables for new runs, command-line arguments allow a new FIREFLY pipeline to be completly flexible and scalable. argparse is a widely used Python library and proves very effective when creating quick and easy-to-use wrappers for FIREFLY.

# Import argparse into your new launch wrapper
import argparse

# Define some command-line arguments (example that could be added to firefly.py)
parser = argparse.ArgumentParser(description="Run FIREFLY on a single spectrum")
parser.add_argument("--input", required=True, help="Input spectrum file")
parser.add_argument("--output", required=True, help="Output spFly FITS file")
parser.add_argument("--spec-index", type=int, default=None, help="Index for multi-object")
args = parser.parse_args()

# args.input, args.output, args.spec_index can now be passed trough the pipeline

Example usage:

# Single spectrum
python firefly_WRAPPER.py --input spectrum.fits --output spFly-spectrum.fits

# Multi-object FITS file
python firefly_WRAPPER.py --input desi_chunk.fits --spec-index 42 --output spFly-042.fits

Implementing Parallel Processing - SLURM (HPC cluster execution)

On HPC systems, the SLURM scheduler controls allocations of resources for your submitted jobs and so just like the DESI pipeline custom SBATCH scripts can be created for your new FIREFLY launch wrapper. Inputs, output, indices and configuration variables can then also be parsed through the pipeline from the SBATCH script.

#!/bin/bash
#SBATCH --job-name=firefly
#SBATCH --cpus-per-task=32
#SBATCH --time=04:00:00
#SBATCH --output=firefly_%j.log

export FF_MAX_WORKERS=$SLURM_CPUS_PER_TASK

# Pass input, output, and optional spec index to the Python script
python firefly_WRAPPER.py \
  --input new_survey.fits \
  --output spFly_new_results.fits \
  --spec-index 42

Or process a range of spectra in a loop:

#!/bin/bash
#SBATCH --job-name=firefly_array
#SBATCH --cpus-per-task=32
#SBATCH --array=0-999%10
#SBATCH --time=04:00:00
#SBATCH --output=firefly_%a.log

export FF_MAX_WORKERS=$SLURM_CPUS_PER_TASK

# Use SLURM_ARRAY_TASK_ID to process different spectra in parallel jobs
SPEC_INDEX=$SLURM_ARRAY_TASK_ID
python firefly_WRAPPER.py \
  --input new_survey.fits \
  --output spFly_${SPEC_INDEX}.fits \
  --spec-index $SPEC_INDEX

These are some examples to scale up FIREFLY on new data using parallel processing and parse arguments. But providing FIREFLY is given the required spectral inputs and metadata, custom wrappers can be made for any new galaxy data sources. As seen from the various survey pipelines FIREFLY provides, the code can be tailored and expanded to explore different spectral data on a large-scale.

Tips & Best Practices

• Verify units and calibration: ensure flux is in 10^-17 erg/s/cm²/Å and wavelengths are consistently vacuum or air as expected.
• Check redshift-dependent age limits: FIREFLY constrains template ages by the spectrum's redshift, be sure to adjust this if you want to use a different cosmology.
• Monitor job status: use squeue --me in the HPC terminal to check running/pending SLURM jobs and scancel to abort if needed.
• Organise outputs: direct each array job to its own folder or filename prefix (like the DESI pipelines) to avoid any overwriting and clarity between different runs.
• If resources are tight, adjust HPC memory allocations and lower spectra per job or reduce dust smoothing to speed up fitting.