Release notes

Version 0.15.0

This release requires openMINDS 0.6.1 (openminds>=0.6.1).

openMINDS v5 support

fairgraph now supports openMINDS v5 alongside v4, on an experimental basis. Both sets of classes are available at the same time, and v4 remains the default, so existing code continues to work unchanged:

import fairgraph.openminds.core as omcore          # v4 (default)
import fairgraph.openminds.v4.core as omcore4      # explicit v4
import fairgraph.openminds.v5.core as omcore5      # explicit v5

v5 covers the same metadata domains as v4, plus a new neuroimaging domain (openminds.v5.neuroimaging) for MRI acquisitions and the devices used to make them. A number of classes have been renamed — for example BrainAtlas is now AnatomicalAtlas, and CommonCoordinateSpace is now CommonCoordinateFramework — and others have been added. See Metadata domains for the full list.

A v4 class and its v5 counterpart share the same node type URI in the Knowledge Graph, so a response cannot be assigned to a version by inspecting it. The client therefore has to be told which version to deserialize into:

from fairgraph import KGClient
import fairgraph.openminds.v5.core as omcore

client = KGClient(host=host_serving_v5_metadata, openminds_version="v5")
people = omcore.Person.list(client)

openminds_version accepts "v4" (the default) or "v5"; anything else raises ValueError. Use classes from the same version as the client you pass them to.

Warning

v5 support is experimental in this release.

The migration of the EBRAINS Knowledge Graph to openMINDS v5 is still under way. The production and pre-production deployments serve v4; v5 metadata is so far only available from a development deployment with restricted access, whose contents are incomplete and liable to change. The v5 support in this release has therefore not yet been exercised against a fully populated KG.

Until that changes, the v5 classes and the openminds_version argument may be altered in backwards-incompatible ways in any release, without the deprecation period that applies to the rest of the API. openMINDS v4 support is unaffected and continues to follow the normal compatibility rules.

Searching by name

by_name() is now at parity with the openMINDS method it overrides. If no client argument is provided, the function calls the underlying openMINDS method, searching within the builtin openMINDS instance library and returning objects with semantic IRIs. If a client is provided, the function searches the Knowledge Graph and returns objects with UUID-based ids.

Both paths now accept match="within" (the search string contains the name), case_sensitive (default True, as in openMINDS) and ignore_accents (which also treats special letters as their plain-letter equivalents, e.g., ß as ss, œ as oe).

The Knowledge Graph search also now covers all the name-like properties a class has, rather than stopping at the first. It previously searched only one, which is why, for example, License.by_name("CC-BY-NC-4.0", client) found nothing: full_name was searched and short_name was not (#131).

Regular-expression filters

Filters may now be given as a Regex instead of a plain string, which selects the KG’s “REGEX” operator in place of “CONTAINS”:

from fairgraph import Regex

people = Person.list(client, family_name=Regex("^M[uü]ller$"))

Note that searches are always case-insensitive.

Saving objects that already exist in another space

save() now looks for an existing matching instance in all spaces, not just the one it is about to write to. Previously, a locally-constructed object whose counterpart lived in a different space was not recognized, and a duplicate was created. This happened most often with recursive=True, where new child objects take on the parent’s target space (#136).

If a match is found in the target space, it is used. If the only match is in another space, it is updated in the space where it lives, with a warning, rather than being duplicated. Multiple matches, none of them in the target space, raise an exception unless ignore_duplicates=True.

Removals

The openMINDS v3 transitional machinery has been removed. The KG has been serving v4 metadata for some time, and the code that translated between the v3 and v4 namespaces is no longer needed. The following have been removed:

  • KGClient.migrated, the feature-detection probe that decided at run time whether the KG a client was connected to had been migrated to v4;

  • the fairgraph.utility functions adapt_namespaces_3to4, adapt_type_4to3, adapt_namespaces_4to3, adapt_namespaces_for_query and types_match.

Code that called these directly will need updating; code that simply used the client is unaffected.

Changes in behaviour

Three changes come with the rework of by_name():

  • searching with a client on a class that has no name-like property at all (DOI, ORCID, WebResource, …) now raises AttributeError;

  • an unrecognized match argument now raises ValueError instead of being silently treated as "contains";

  • the no-client path no longer warns when several objects share a name, since openMINDS does not. The warning remains on the Knowledge Graph path.

Other changes:

  • space_info() no longer raises an error when a space contains types that have no fairgraph class, such as the KG’s own internal types. These now appear in the result keyed by their type IRI (a string) rather than by a class. The ignore_errors argument, which is no longer needed, has been removed (#113).

  • query() now raises ValueError if a value in its filter argument contains “+” or “%”. These values are sent to the KG as request parameters, which the KG does not decode correctly, so the query would otherwise silently return the wrong results. Filters given to list() and similar methods are not affected.

Bug fixes

  • Saving the same object twice in one session no longer erases the properties it does not carry. (#134).

  • by_name() no longer raises TypeError when called without a client and nothing matches; it returns None. (#130).

  • KGObject instances are now hashable, so they can be used in sets and as dictionary keys. Defining __eq__ had made them unhashable; they now hash by identity, as the openMINDS classes do.

  • from_id() now returns None for a non-existent id when called on the base KGObject class (i.e., when the type is not known in advance), instead of raising TypeError (#115).

  • An instance created by save() is now recorded in the save cache. Since the KG is only eventually consistent, a newly-created instance may not yet appear in existence queries, so saving a second, equivalent object in the same session could previously create a duplicate (#137).

  • exists(), and therefore save(), no longer raises TypeError for new objects of classes whose existence query includes a date or datetime property, such as Book (#139).

  • Filter values containing “+” can now be matched. fairgraph previously cut string filter values off at the first “+” (or raised ValueError if it came within the first three characters), to work around a KG bug. That bug only affects filter values sent as request parameters, which fairgraph no longer does, so values such as "C++", email addresses with a “+” suffix, or timestamps with a UTC offset are now searched for in full.

  • Where exists() hit a connection error other than a dropped connection, it raised a NameError; the original error is now re-raised.

  • Links to other KG instances can now be resolved when they are given as bare UUIDs rather than full URIs. Version 4 of Marmotgraph, the software behind the Knowledge Graph, gives links in this form; it is already used by the development deployment serving openMINDS v5 metadata. KGClient now expands them to full URIs, so that links can be resolved and compared with the ids of the instances they point to (#144).

Documentation

The developer documentation in Contributing to fairgraph has been expanded (#73). It now covers regenerating the v4 and v5 classes together, the hand-written methods that are merged into generated classes, running the tests with and without a KG token, and the versioning scheme. It also points to the issue-tracker milestones as the project roadmap.

Version 0.14.0

This release requires openMINDS 0.6.0 (openminds>=0.6.0) and inherits two changes in behaviour from it.

Links between library instances now give you objects, not dictionaries. The openMINDS instance libraries (available as class attributes, e.g. Species.mus_musculus) contain many cross-references: an atlas region points to its versions, a parent region to its children, and so on. Until now those references came back as plain {"@id": ...} dictionaries, so there was nothing useful you could do with them without a separate lookup. They are now the actual objects:

>>> from fairgraph.openminds.sands import ParcellationEntity
>>> region = ParcellationEntity.aal1_acin

# up to 0.13.6, region.has_versions[0] was a plain dict:
{'@id': 'https://openminds.om-i.org/instances/parcellationEntityVersion/AAL1_SPM12-v4_ACIN'}

# from 0.14.0 it is a ParcellationEntityVersion, which you can use directly:
>>> region.has_versions[0].version_identifier
'SPM12, v4'
>>> region.has_versions[0].name
'anterior cingulate and paracingulate gyri'

If your code worked around the old behaviour by reading the "@id" key out of these dictionaries, it will need updating. References that point outside the instance libraries are still KGProxy objects, resolved with resolve() as before.

Invalid IRIs are now reported at validation time, not on creation. IRI("not a url") used to raise ValueError immediately. It now succeeds, and the problem is reported when the containing object is validated, together with any other validation failures, and following whatever error handling you have configured (see fairgraph.set_error_handling()):

>>> repo = FileRepository(name="my repository", iri=IRI("not a url"), hosted_by=organization)
defaultdict(<class 'list'>, {'value': ['Invalid IRI']})
>>> repo.validate()
{'value': ['Invalid IRI']}

Code that relied on catching ValueError around IRI creation should check the validation result instead.

Other changes in this release:

  • import fairgraph works again with openMINDS 0.6.0. Preparing the instance libraries used to walk the links between instances, which the newly-connected atlas instances turned into an endless loop (a RecursionError, or exhausted memory); it no longer does (#116).

  • Fixed an out-of-date data-proxy URL that made download() fail for some datasets, models and atlases (DatasetVersion, ModelVersion, BrainAtlasVersion, CommonCoordinateSpaceVersion).

  • fairgraph.openminds.set_error_handling() now sets error handling for every openMINDS submodule in a single call.

Version 0.13.6

Bug fixes in this release:

  • Fixed a KeyError when calling query() with both instance_id and a custom id_key on queries using a custom responseVocab (#107).

Version 0.13.5

Bug fixes in this release:

  • Fixed a bug where save() could silently no-op after re-fetching an object via from_id(), because KGClient.cache was not invalidated by writes (#110).

  • Fixed the repository IRI used by the dataset version download() method (#109).

Version 0.13.4

Bug fixes in this release:

  • Fixed a TypeError when trying to filter by parts of an IRI.

  • Fixed get_journal() when the journal is a KGProxy (issue + volume case).

Other changes:

  • Added GDPR documentation.

Version 0.13.3

Main changes in this release:

  • Improved robustness of Collection.upload() (automatic retry on failure, token refresh, and correct resume behaviour).

  • In the query-builder module (fairgraph.queries), added a new PathElement class, which allows reverse and type_filter to be set independently on each step of a multi-element query path. Previously, reverse and type_filter applied only to the first element of the path.

Here is an example of using PathElement to find all files belonging to a particular dataset by traversing a fileRepository link forward and a repository link in reverse, filtered to DatasetVersion nodes:

from fairgraph.queries import Query, QueryProperty, Filter, PathElement

query = Query(
    node_type="https://openminds.om-i.org/types/File",
    properties=[
        QueryProperty(
            [
                "https://openminds.om-i.org/props/fileRepository",
                PathElement(
                    "https://openminds.om-i.org/props/repository",
                    reverse=True,
                    type_filter="https://openminds.om-i.org/types/DatasetVersion",
                ),
                "@id",
            ],
            name="dataset",
            expect_single=True,
            filter=Filter("CONTAINS", value="<dataset-uuid>"),
        ),
        QueryProperty("https://openminds.om-i.org/props/name", name="name"),
    ],
)
response = client._kg_client.queries.test_query(
    payload=self.serialize(),
    stage=Stage.RELEASED,
    pagination=Pagination(start=0, size=5)
)

Also fixed a bug where calling .exists() on a ModelVersion with a repository that had no @id raised an error.

Version 0.13.2

Fixed some bugs in the machinery for supporting the openMINDS v3 to v4 migration. This version has been tested with both unmigrated (v3) and migrated (v4) deployments of the KG.

Version 0.13.1

Fixed a bug where checking if an object exists was resetting any properties that had been changed to None locally.

Version 0.13.0

For this version we have extensively rewritten fairgraph, to build directly on the openMINDS Python library. This

  • ensures (almost) perfect compatibility between the openMINDS API and the fairgraph API, so people can start developing locally with openMINDS-Python, then just change to importing “fairgraph.openminds” instead of “openminds.v4” when they wish to upload metadata to the Knowledge Graph.

  • adds functionality for working with local JSON-LD files in fairgraph.

  • provides the openMINDS instances libraries as class attributes, e.g., Species.mus_musculus

  • adds the Collection, which has the functionality of the equivalent class in openMINDS-Python, but in addition has support for uploading an entire metadata collection to the KG in a single call.

The documentation has been refreshed and extended.

This version of fairgraph provides the openMINDS v4 schemas.

There is one breaking change, the keyword argument “scope” has been renamed to “release_status”.

Version 0.12.2

This version includes various small changes and bug fixes.

  • Improvements to client.user_info() implementation, to give better error messages in case of failure.

  • Use numbers.Real in place of float in fields where openMINDS specifies type “number”.

  • More informative error message for unserializable values.

  • Optimization of the exists() method, to improve performance by using a simpler query.

  • Allow restricting exists() to specific spaces.

  • Don’t allow creating IRI objects with an empty string.

Version 0.12.1

This version includes various small changes and bug fixes.

  • Reverse properties/fields, introduced in version 0.11, are now contained in the attribute reverse_properties, which makes it easier to operate only on intrinsic, “forward” properties or only on reverse properties.

  • The type_ attribute is now a string (as in openMINDS Python), rather than a single-element list containing that string (as in the KG).

  • The KGClient now allows interactive authentication via a web-browser (“device flow”), if the client cannot find an authorization token. To disable this (for example for non-interactive use), create the client with allow_interactive=False.

  • Added methods space_info() and clean_space() to KGClient, to make it easier to clean up private and collab KG spaces used for testing and development.

  • Added method move_all_to_space() method to KGClient, primarily intended for use by KG curators.

  • Added the option to ignore duplicates in node.exists() calls (by default, if multiple nodes match the existence query an Exception will be raised)

  • Updated the openminds module to include the latest changes in the openMINDS schemas.

  • Partially fixed a bug in which resolving certain reverse links is broken.

Version 0.12.0

The main change in this release is the harmonization of terminology between fairgraph and the openMINDS Python library. In particular, Field/fields has been changed to Property/properties, but there are also various other changes. The previous names should still work, and should emit a deprecation warning. The previous names will be removed in a future release.

In addition, the openminds module builder has been rewritten using the new openMINDS build pipeline. This should not have resulted in any substantive changes to the released fairgraph package.

Version 0.11.0

“Reverse” fields

fairgraph now includes “reverse” fields, i.e. fields that connect to objects that aren’t defined directly in the schema for a given type A, but are defined in other types, B, C that link to objects of type A. To take an example, Model has a field versions, which links to objects of type ModelVersion. Now we’ve added to ModelVersion a “reverse” field is_version_of, which links back to the Model.

These reverse links can be resolved, and can be used for queries. For example, if you are starting from a ModelVersion, and wish to find its associated Model, previously you had to perform a query:

>>> models = omcore.Model.list(client, versions=model_version)
>>> model = models[0]

Now, you can just resolve the reverse field:

>>> model = model_version.is_version_of.resolve(client)

The original method also still works, and could be more efficient, depending on how many objects of each type there are. If performance is an issue, it is best to profile both approaches.

Perhaps more usefully, you can now ask fairgraph to resolve the Model at the moment of obtaining the ModelVersion, e.g.

>>> model_version = omcore.ModelVersion.from_id(
...     "5c52380c-7bd9-4fe6-8d72-ff340250b238",
...     client,
...     follow_links={"is_version_of": {}}
... )
>>> type(model_version.is_version_of)
<class 'fairgraph.openminds.core.products.model.Model'>
>>> model_version.is_version_of.uuid
'be001074-7eab-4c7e-9bde-9e5987b085d2'

and you can also make queries across these reverse links, e.g.

>>> model_versions = omcore.ModelVersion.list(
...     client,
...     is_version_of="be001074-7eab-4c7e-9bde-9e5987b085d2"  # id of a Model
... )
>>> model_versions[0].uuid
'5c52380c-7bd9-4fe6-8d72-ff340250b238'

Note

reverse links that pass via EmbeddedMetadata instances are not yet supported. For example: SoftwareVersion has a field copyright, which contains embedded metadata of type Copyright (which does not have its own ID). Copyright has a field holders which links to Person, among others. At present, it is not possible to access the SoftwareVersion from a Person by way of a reverse field, since the link is not direct. (You can still make a forward query, though). Such indirect reverse fields will be implemented in a future version of fairgraph.

Other changes

  • made the follow_links argument to resolve() behave the same way as for list(), from_id(), etc., i.e. it expects a structure of nested dicts to specify explicitly which links to follow, rather than an integer meaning “follow all links for this number of levels”.

  • added set_error_handling() as a module-level function, so you can control the behaviour of all classes in a module (e.g. fairgraph.openminds.core) in a single line.

Version 0.10.0

New/modified functionality

  • more flexible “strict_mode” - replace [True, False] with Enum[“error”, “warning”, “log” none”], rename to “error_handling”, and make ErrorHandling.log the default

  • support filters that cross links in the graph

  • implement more fine-grained control of specifying links to follow when creating queries

  • add “follow_links” argument to from_uri(), from_uuid, from_id, from_alias and by_name

  • remove “resolved” keyword argument and replace with “follow_links”

  • improve “queries” module to expose more of the available features of the API

  • allow KGObject.from_id() to work with cls=KGObject, i.e. when we have an @id but don’t know its type

  • add an __init__() method with explicit field names to all KGObject sub-classes, to catch incorrect keyword arguments

  • rename “type” class attribute to “type_” to avoid clashing with “type” as an openMINDS property name

  • regenerate fairgraph.openminds based on latest openMINDS v3-dev

  • remove mention of “v3” from module and variable names

  • remove code relating to KG v2

Code/documentation quality

  • update documentation - added developers’ guide and code-of-conduct

  • add codemeta.json

  • code cleanup and refactoring

  • add docstrings to most classes and methods that were missing them

  • formatted codebase with black

  • started adding type annotations

  • deserialization of EmbeddedMetadata uses the same machinery as KGObject

  • simplify internal data handling (in particular detecting updated fields).

  • remove unused code

  • switch to using expanded keys (URIs) in KGObject.data, to reduce the risk of confusion, since the KG always returns data with expanded keys.

  • make expand_uri consistent with compact_uri in how it handles single uris vs lists of uris

  • remove dependency on pyld

  • by default, don’t use stored queries, use the latest generated ones

  • more unit tests

Version 0.9.0

  • implement the “match” argument of the by_name() method

  • change configure_space() to take the space name, not the collab id, as its argument

  • fix DatasetVersion.download() for unreleased data repositories

  • better handling of the scenario when self.exists() gives the wrong answer, so we get an error on creating a new instance

  • distinguish authorization and authentication errors, and allow being more forgiving with authorization errors

  • fix some bugs when using fairgraph without curator privileges

  • add “allow_update” attribute to KGObject (True by default), to support preventing attempted updates when needed

  • more informative error messages

  • better handling of the situation where fields with multiple=False receive multiple items

  • when calculating which fields need to be updated, handle expanded and compacted paths

  • better documentation of controlled terms, including adding a list of possible values and ontology links to docstrings

  • switch to building project with pyproject.toml

  • update openMINDS schemas

Version 0.8.2

  • more informative error message when failing to generate cache key

  • add KGClient method to move instances between spaces

  • allow client.query(…, scope=”any”,…) to work with custom queries (ones not generated by fairgraph)

  • add scope=”any”

  • update openMINDS schemas, including adding “chemicals” extension

  • add “instance_id” option to kgclient query() method

Version 0.8.1

  • recursive save now handles EmbeddedMetadata objects that _contain_ KGObjects (e.g. QuantitativeValue→UnitOfMeasurement, Affiliation→Organization)

  • space no longer defaults to the class default

  • make it clear that data and space are required for create_new_instance()

  • fix release()/unrelease() methods, and add support for recursive releasing (i.e. following tree of children)

Version 0.8.0

  • update to work with new ebrains-kg-core package release (from PyPI)

  • add configure_space(collab_id, types) method to KGClient

  • updates following recent openMINDS schema changes

  • avoid confusing error messages when importing fairgraph if kg-core-python not installed

Version 0.7.1

  • run tests with Github Actions

  • fix a few bugs

Version 0.7.0

  • add download() methods

  • support use of KGProxy objects as filter values

  • updates to reflect recent changes in openMINDS

  • more flexibility in delete() method

  • store the scope from which an object was queried

  • add from_alias()

  • if unable to store queries to the preferred space, use “myspace”

  • prevent writing to “controlled” space

  • assorted bug fixes

  • cleaner separation between KGObject and KGClient functionality

  • handle lists of filter values

  • add a “follow_links” argument to the resolve() methods, to avoid having to manually resolve links.

  • order fields in openMINDS classes alphabetically, except for certain priority fields that act as unique names

  • refactor queries to allow dynamically generated queries based on filter settings, not only previously-stored queries

  • move fairgraph openMINDS generator from openMINDS_generator to fairgraph repository

  • change default strict mode to False

  • make v3 the default

  • add support for typeFilter in queries, and use this to re-enable support for cases where different allowed classes have different fields, such as QuantitativeValue and QuantitativeValueRange for age, weight

  • make pyxus and openid_http_client optional dependencies, so people using only KGv3 can install fairgraph without them

  • add documentation of openMINDS classes

Version 0.6.0

  • support for openMINDS and KG v3

  • improved handling of spaces when saving

  • handle serialization of KGProxy objects

  • added “replace” option to KGObject.save(), and implemented client.delete_instance() and client.replace_instance()

  • add CI testing with Python 3.9

  • handle expiring tokens better, since kg_core_python doesn’t consider 401 and 403 responses as errors

  • add queryable logging of activity when saving, to help debug problems with KG updating

  • when saving recursively, non-top-level objects that already exist in a space are updated in that space, and existing controlled terms are not updated.

  • raise a NameError if unrecognized keyword arguments are based to a KGObject constructor, helps avoid misspellings passing unnoticed.

  • add caching of queries, to avoid repeated network requests

  • fix inconsistency in signatures of “resolve()” methods

  • explictly use “latest” scope when getting data while saving

  • support new KG authentication method

  • many new v2 schemas, including live papers, computational provenance, optophysiology

  • update openminds module with latest schemas

  • add utility methods Person.me() and File.from_local_file()

  • add “from_index” argument to KGQuery.resolve()

  • add “count()” method to KGQuery

  • add the option to load SPDX licence data from a local file rather than downloading from Github

  • remove Python 2 code

  • drop testing for Python 2.7 and 3.5, add testing for 3.8.

  • can now filter on datetime fields.

  • fix for when query values contain non-ascii characters

  • when updating an object, also update the cached version

  • more robust download method for Dataset