Enhanced Data Masking

Along with the default masking engine provided by the open source edition, Fides for Enterprise customers enables NLP and regex-powered masking strategies to safeguard data at scale. These masking strategy can be used on their own or combined to identify and replace personal information in free-text fields.

The available Enterprise masking strategies are an extension of the open source masking engine. To review standard implementation options, see the open source documentation.

NLP data masking#

Natural language processing, or NLP, allows Fides to perform sophisticated de-identification of personally identifiable information embedded in free-text. This strategy uses a machine learning model trained specifically to recognize and redact personal information that is included as part of free-text. The redacted text areas will be replaced with generic text (e.g. [NAME_1]).

As with the open source masking strategy, NLP data masking can be invoked by calling the /mask endpoint.

PUT /api/v1/masking/mask:

{
    "values": ["test@example.com", "my name is john"],
    "masking_strategy": {
        "strategy": "nlp",
        "configuration": {
            "multilingual" : false
        }
    }
}

Successful calls to /mask will return a version of the input with the masked values:

{
    "plain": [
        "test@example.com",
        "my name is john"
    ],
    "masked_values": [
        "[EMAIL_ADDRESS_1]",
        "my name is [NAME_1]"
    ]
}

Configuration#

Accuracy modes#

The NLP service provides four accuracy modes to use when processing input text. The supported modes are fast, standard, standard_high and high.

These modes allow for tradeoffs between throughput (speed) and model accuracy, with fast being the fastest mode and high being the most accurate mode.

If no accuracy mode is specified, the accuracy mode will default to fast when multilingual is set to true. Otherwise, the default accuracy mode is standard.

The mode fast is only available when multilingual is set to true.

Regex data masking#

The regex_replace strategy masks personal information in a given text field by first identifying the text’s language, then evaluating the text against customer-supplied, language-specific regular expressions. Any matching results will be replaced with the customer-specified replacement text (e.g. [NAME_1]).

If there are no regular expressions defined for the identified language, or the language is identified with low confidence, the entire text field will be replaced by a default.

As with the open source masking strategy, NLP data masking can be invoked by calling the /mask endpoint.

PUT /api/v1/masking/mask:

{
   "values":[
      "tall male suspect",
      "1 N Albany Cir",
      "patient blood type is AB positive",
      "VA, 20402",
      "asdf",
      "berros"
   ],
   "masking_strategy":{
      "strategy":"regex_replace",
      "configuration":{
      }
   }
}

Successful calls to /mask will return a version of the input with the masked values:

{
    "plain": [
        "tall male suspect",
        "1 N Alabany Cir",
        "patient blood type is AB positive",
        "VA, 20402",
        "asdf",
        "berros"
    ],
    "masked_values": [
        "tall gender suspect",
        "street",
        "patient blood type is AB positive",
        "MASKED",
        "MASKED",
        "MASKED"
    ]
}

Configuration#

Fides supports defining your regex_replace configuration as YAML. For more information, see how to supply a configuration file below.

Using a configuration file#

Due to the potential length of the regular expression lists, universal_regex and language-based regular expressions should be specified in a fides.toml file under the `regex_replace. This file may be supplied at container runtime via volume mount.

Universal regular expressions#

Universal regular expressions should be defined under a universal_regex key as a list of lists, containing pairs of regular expression patterns followed by the replacement text:

[regex_replace]
universal_regex = [
    ["<regular expression pattern>", "replacement text"],
    ["<regular expression pattern>", "replacement text"]
]

Other regular expressions#

Regular expressions should be defined by language code, as a list of lists, and contain pairs of regular expressions followed by replacement text. Regular expression search is case-insensitive.

[regex_replace]
EN = [
      ["concert|movie|snorkeling class|art exhibit", "[EVENT]"],
]

In this example, if the language of your value was identified as EN, the words concert, movie, snorkeling class, or art exhibit would be replaced with the text [EVENT].

Combined masking strategies#

To maximize accuracy in identifying personal information among free-text, you may optionally specify both the nlp strategy and the regex_replace strategy in a single request. The strategies will be executed in the order specified.

In the below example, the values are run against the NLP engine before being run against the regular expression strategy. This example also adds the required_confidence and default_mask parameters to the nlp strategy.

{
   "values":[
      "tall male suspect",
      "1 N Alabany Cir",
      "patient blood type is AB positive",
      "VA, 20402",
      "asdf",
      "berros"
   ],
   "masking_strategy":[
          {
         "strategy":"nlp",
         "configuration":{
            "multilingual": true,
            "required_confidence": 0.25
         }
      },
   
      {
         "strategy":"regex_replace",
         "configuration": {
            "required_confidence": 0.25,
            "default_mask": "MASKED"
         }
      }
   ]
}

Response:

{
    "plain": [
        "tall male suspect",
        "1 N Albany Cir",
        "patient blood type is AB positive",
        "VA, 20402",
        "asdf",
        "berros"
    ],
    "masked_values": [
        "tall gender suspect",
        "addr:street_num",
        "patient blood type is [BLOOD_TYPE_1]",
        "MASKED",
        "MASKED",
        "MASKED"
    ]
}

Supported languages#

The following languages are supported by the NLP masking engine when multilingual is set to true. Only English is supported when multilingual is set to false.

Core support indicates the language is fully supported. Extended support indicates a “beta” level support that may require further optimizations.