PyNX: Python tools for Nano-structures Xtallography#

Introduction#

PyNX stands for Python tools for Nano-structures Xtallography. It can be used for:

  • Coherent X-ray imaging simulation and analysis: coherent diffraction imaging (CDI), Ptychography, Wavefront propagation, near field and far field techniques…

  • Fast scattering calculations from large number of atoms and reciprocal space positions.

PyNX is fully optimised to use Graphical Processing Units, using either CUDA or OpenCL, to provide fast calculations with 1 to 3 orders of magnitude speedup compared to standard processor calculations.

NEWS#

  • 2025/02/09: PyNX 2025.1 is available. See the full Changelog

  • 2024/06/29: PyNX 2024.1.1 is available.

  • 2024/06/11: PyNX 2024.1 is available.

  • 2023/02/03: PyNX 2023.1 is available.

  • 2022/02/06: PyNX 2022.1 is available.

  • 2020/12/12: PyNX 2020.2.2 is available with a few fixes.

  • 2020/10/23: PyNX 2020.2 is available, notably introducing distributed computing on multiple GPU

  • 2020/09: the PyNX article: PyNX: high-performance computing toolkit for coherent X-ray imaging based on operators is out: J. Appl. Cryst. 53 (2020), 1404, also available as arXiv:2008.11511

  • 2020/02/02: version 2020.1 is out !

  • 2019/06/19: version 2019.2.6 is out. Note that 2019.2.x (x<6) versions had incorrect Ptycho CUDA scaling, preventing correct ML minimisation

  • 2018/06: all coherent imaging modules have now been converted to use an operator-based API

  • 2016/09: overhaul of the library structure - pynx.gpu has now become pynx.scattering, with pynx.scattering.gid as a sub-module.

Download & Installation#

PyNX is available from:

PyNX can be installed using pip - follow the instructions to install it with all dependencies in a (conda) python environment.

PyNX usage#

PyNX scripts#

PyNX can be used simply with command-line scripts for some applications (2D/3D CDI and 2D Ptychography). These can take generic files as input, such as CXI files (http://cxidb.org), or can analyse data directly from beamlines.

PyNX as a python toolkit#

PyNX can be used as a python library with the following main modules:

  • pynx.cdi: Coherent Diffraction Imaging reconstruction algorithms using GPU for Coherent Diffraction Imaging,

    in 2D or 3D, for small-angle or Bragg diffraction data. This uses either CUDA or OpenCL, but CUDA is strongly recommended for 3D data (significant speedup).

  • pynx.holotomo: Holo-tomography reconstructions (object and illumination) algorithms using GPU (CUDA).

  • pynx.ptycho: simulation and analysis of experiments using the ptychography technique, using GPU (OpenCL, CUDA).

    Examples are available in the pynx/examples directory. Scripts for analysis of raw data from beamlines are also available, as well as using or producing ptychography data sets in CXI (Coherent X-ray Imaging) format.

  • pynx.scattering: X-ray scattering computing using graphical processing units, allowing up to 2.5x10^11 reflections/atoms/seconds

    (single nVidia Titan X). The sub-module``pynx.scattering.gid`` can be used for Grazing Incidence Diffraction calculations, using the Distorted Wave Born Approximation

  • pynx.wavefront: X-ray wavefront propagation in the near, far field, or continuous (examples available at the end of wavefront.py).

    Also provided are sub-modules for Fresnel propagation and simulation of the illumination from a Fresnel Zone Plate, both using OpenCL for high performance computing.

Changelog#

See the full Changelog:

Changelog#

Version 2025.1.1 (2025-02-XX)#

  • CDI:

    • Add smoothing/TV regularisation parameter for ER, HIO, RAAR following Malm & Chushkin, J. Synchrotron Rad. 32 (2025), 210

  • Ptycho: * Fix for pynx-ptycho-hermes [F. Picca]

  • Documentation:

    • Add pynx-cdi-simulation

Version 2025.1 (2025-02-09)#

  • CDI:

  • Holotomo:

    • fix –prefix_output with –save_edf and an absolute path

    • fix run number increase and directory creation when using –processed_data [https://gitlab.esrf.fr/favre/PyNX/-/issues/205]

    • add Nabu’s octave-accurate method for the CoR

    • add checks for the results of CoR

    • More reliable half-tomo reconstructions using Nabu 2024.2

    • allow to save directly into processed data folders (ESRF)

    • preliminary support for bliss/nexus data format

  • Ptycho:

    • add more default values which can be changed in derived runners

    • deprecated –dark_subtract (no scheduled EOL yet)

    • Update Hermes runner (F. Picca)

    • pynx-ptycho-analysis: direct exit (code 2) when missing parameters

    • correct opencl device selection with MPI (https://gitlab.esrf.fr/favre/PyNX/-/issues/206)

  • Small fixes for numpy>=2 (some dependencies still prevent using numpy>=2 as of this release)

  • fix OpenCL/CUDA tests exclusion handling

Version 2024.1.1 (2024-06-29)#

  • CDI:

    • correct --roi handling for scripts

    • correct --modes_output interpretation in pynx-cdi-analysis

  • Scattering: fix CPU calculations using SSE code on x86.

  • Other: Replace scipy.integrate.cumtrapz() by cumulative_trapezoid()

Version 2024.1 (2024-06-11)#

  • General: switch all command-line scripts formatting to argparse, changing e.g. “pynx-cdi-xx data=data.h5” to “pynx-cdi-xx --data=data.h5” or equivalently “pynx-cdi-xx --data data.h5”. Old format of command-lines are deprecated but will be automatically converted for backwards compatibility until the next release.

  • documentation is now available from https://pynx.esrf.fr

  • CDI:

    • enable mask interpolation for 2D data

    • added a pynx-cdi-simulation script for testing

    • [BUG] CDI: during mode analysis only for 2D CDI solutions, correctly test for the conjugate twin solution alignment [https://gitlab.esrf.fr/favre/PyNX/-/issues/193]

  • Ptycho

    • ID16A NFP script: add an option to save directly the object’s derived phase. Save NXsample metadata. Add nxtomo export, save angle for ptycho-tomo.

    • add the ability to load an object and/or probe array from a file for pynx-ptycho-simulation

    • allow some mixing of AP within DM to stabilise the reconstruction (inspired by ptypy feature, different implementation)

    • Add RAAR operator

    • save some extra memory during DM to compute LLK

    • add shared probe option to loop over algorithm steps while either sharing the same probe or averaging/reducing it, every N cycles. This can be used when optimising different scans which are swapped in/out GPU memory periodically.

    • scripts: correct an issue when using –rebin (which could result in an incorrect pixel and object size)

    • scripts: add –saveplot=object_real, object_imag and object_abs

    • scripts: avoid issue with history statistics when using verbose=0

    • scripts: add ability to load script parameters from a file

  • Holotomo:

    • first release with (beta) support for holo-tomography reconstructions

    • main features include: phasing based on CTF or Paganin, followed by iterative algorithms (AP mainly, DM, DRAP and RAAR for testing), either on normalised data or by optimising the illumination from multiple projections and multiple propagation distances, padding, distortion correction, alignement, double-flat-field correction,..

    • Includes the pynx-holotomo-id16b script for ESRF ID16B data, which handles alignment of multiple distances, and tomography reconstruction thanks to the tomotools and nabu projects (https://gitlab.esrf.fr/tomotools)

    • Partly based on prior developments from the ID16A beamline (see notably https://doi.org/10.1364/OE.26.011110 and https://doi.org/10.1063/1.125225 )

  • scattering:

    • CPU calculations have been corrected, and will revert to numpy on non-x86_64 architectures or when cython is not available

    • an error will be raised if a GPU is requested and none can be selected.

Version 2023.1.1 (2023-02-05)#

  • fix multiple display of figures using %matplotlib widget (ipympl)

  • update requirements

Version 2023.1 (2023-02-03)#

Major changes#

  • command-line scripts have been renamed to be more standard and without a .py extension, e.g. pynx-ptycho-cxi instead of pynx-cxiptycho.py

  • Ptycho (near field): added Paganin operator to estimate the initial object, using a reference frame with the direct beam

  • Ptycho: if a reference frame (direct beam) is given, it can be also used for optimisation, yielding an absolute reference for the probe.

  • Ptycho (API): the intensity mask should not be supplied anymore, but the iobs data should have masked data as negative values. For near field, padded data must be extrapolated and stored as -1-I_extrapolated.

  • Ptycho: add cxi_output command-line option to e.g. only save the object phase and save space.

  • CDI: correct CUDA random initialisation of object which was not-so-random

Mailing list, git & issue tracker#

There is a mailing list for PyNX: to subscribe, send an email to pynx-subscribe@esrf.fr

To access the git repository and the issue tracker, got to http://gitlab.esrf.fr/favre/PyNX

Citation & Bibliography#

If you use PyNX for scientific work, please consider including a citation:

PyNX re-uses or was inspired by features described in the following articles and open-source software packages:

  • CDI:

    • J. N. Clark, X. Huang, R. Harder, and I. K. Robinson, “High-resolution three-dimensional partially coherent diffraction imaging,” Nat Commun 3, 993 (2012).

    • S. Marchesini, “A unified evaluation of iterative projection algorithms for phase retrieval,” Review of Scientific Instruments 78(1), 011301 (2007).

  • Ptychography:

    • PtyPy: B. Enders and P. Thibault, “A computational framework for ptychographic reconstructions”, Proc Math Phys Eng Sci 472(2196), (2016).

    • M. Odstrčil, A. Menzel, and M. Guizar-Sicairos, “Iterative least-squares solver for generalized maximum-likelihood ptychography,” Optics Express 26(3), 3108 (2018).

    • S. Marchesini, A. Schirotzek, C. Yang, H. Wu, and F. Maia, “Augmented projections for ptychographic imaging,” Inverse Problems 29(11), 115009 (2013).

    • P. Thibault and A. Menzel, “Reconstructing state mixtures from diffraction measurements,” Nature 494(7435), 68–71 (2013).

    • P. Thibault and M. Guizar-Sicairos, “Maximum-likelihood refinement for coherent diffractive imaging” New J. Phys. 14(6), 063004 (2012).

  • Holo-tomograhy:

    • Y. Boliang, L. Weber, A. Pacureanu, M. Langer, C. Olivier, P. Cloetens, & F. Peyrin. “Evaluation of Phase Retrieval Approaches in Magnified X-Ray Phase Nano Computerized Tomography Applied to Bone Tissue”. Optics Express 26, nᵒ 9 (30 avril 2018): 11110.

      1. Cloetens’ (ESRF ID16A) and co-authors octave code for inline holo-tomography

License#

The PyNX library is distributed with a CeCILL-B license (an open-source license similar to the FreeBSD one). See http://www.cecill.info/licences/Licence_CeCILL-B_V1-en.html

Note that CPU computing of the pynx.scattering module uses the sse_mathfun.h header, which is distributed under the zlib license. See http://gruntthepeon.free.fr/ssemath/

See http://ftp.esrf.fr/pub/scisoft/PyNX/README.txt for more details about the license, copyright, as well as other possible issues regarding ptychography.