Release Notes

This page aims to act as a guide for migrating between major SQLFluff releases. Necessarily this means that bugfix releases, or releases requiring no change for the user are not mentioned. For full details of each individual release, see the detailed changelog.

Upgrading to 3.x

This release makes a couple of potentially breaking changes:

  • It drops support for python 3.7, which reached end of life in June 2023.

  • It migrates to pyproject.toml rather than setup.cfg as the python packaging configuration file (although keeping setuptools as the default backend).

  • The serialised output for sqlfluff lint now contains more information about the span of linting issues and initial proposed fixes. Beside the new fields, the original fields of line_pos and line_no have been renamed to start_line_pos and start_line_no, to distinguish them from the new fields starting end_*.

  • When linting from stdin, if there are no violations found - before this version, the serialised response would be simply an empty list ([]). From 3.0 onwards, there will now be a record for the file with some statistics, but the violations section of the response for that file will still be an empty list.

  • The default annotation_level set by the --annotation-level option on the sqlfluff lint command has been changed from notice to warning, to better distinguish linting errors from warnings, which always now have the level of notice. This is only relevant when using the github-annotation or github-annotation-native formats.

  • The previously deprecated --disable_progress_bar on :code:lint, fix and format has now been removed entirely. Please migrate to --disable-progress-bar to continue using this option.

  • The --force option on sqlfluff fix is now the default behaviour and so the option has been deprecated. A new --check option has been introduced which mimics the old default behaviour. This has been changed as it enables significantly lower memory overheads when linting and fixing large projects.

Upgrading to 2.3

This release include two minor breaking changes which will only affect users engaged in performance optimisation of SQLFluff itself.

  • The --profiler option on sqlfluff parse has been removed. It was only present on the parse command and not lint or fix, and it is just as simple to invoke the python cProfiler directly.

  • The --recurse cli option and sqlfluff.recurse configuration option have both been removed. They both existed purely for debugging the parser, and were never used in a production setting. The improvement in other debugging messages when unparsable sections are found means that this option is no longer necessary.

Upgrading to 2.2

This release changes some of the interfaces between SQLFluff core and our plugin ecosystem. The only breaking change is in the interface between SQLFluff and templater plugins (which are not common in the ecosystem, hence why this is only a minor and not a major release).

For all plugins, we also recommend a different structure for their imports (especially for rule plugins which are more common in the ecosystem) - for performance and stability reasons. Some users had been experiencing very long import times with previous releases as a result of the layout of plugin imports. Users with affected plugins will begin to see a warning from this release onward, which can be resolved for their plugin by updating to a new version of that plugin which follows the guidelines.

Templater plugins

Templaters before this version would pass a make_template() callable to the slicing methods as part of being able to map the source file. This method would accept a str and return a jinja2.environment.Template object to allow the templater to render multiple variants of the template to do the slicing operation (which allows linting issues found in templated files to be mapped accurately back to their position in the unrendered source file). This approach is not very generalisable, and did not support templating operations with libraries other than jinja2.

As a result, we have amended the interface to instead pass a render_func() callable, which accepts a str and returns a str. This works fine for the jinja templater (and by extension the dbt templater) as they can simply wrap the original callable with a method that calls render() on the original Template object. It also however opens up the door to other templating engines, and in particular to remote templaters which might pass unrendered code over a HTTP connection for rendering.

Specifically:

  • The slice_file() method of the base templater classes no longer accepts an optional make_template argument or a templated_str argument.

  • Instead a render_func callable should be passed which can be called to generate the templated_str on demand.

  • Unlike the optional make_template - render_func is not optional and should always be present.

Rule plugins

We recommend that the module in a plugin which defines all of the hook implementations (anything using the @hookimpl decorator) must be able to fully import before any rule implementations are imported. More specifically, SQLFluff must be able to both import and run any implementations of get_configs_info() before any plugin rules (i.e. any derivatives of BaseRule) are imported. Because of this, we recommend that rules are defined in a separate module to the root of the plugin and then only imported within the get_rules() method.

Importing in the main body of the module was previously our recommendation and so may be the case for versions of some plugins. If one of your plugins does use imports in this way, a warning will be presented from this version onward, recommending that you update your plugin.

See the Developing Plugins section of the docs for an example.

Upgrading from 1.x to 2.0

Upgrading to 2.0 brings several important breaking changes:

  • All bundled rules have been recoded, both from generic L00X formats into groups within similar codes (e.g. an aliasing group with codes of the format AL0X), but also given names to allow much clearer referencing (e.g. aliasing.column).

  • Rule Configuration now uses the rule name rather than the rule code to specify the section. Any unrecognised references in config files (whether they are references which do match existing rules by code or alias, or whether the match no rules at all) will raise warnings at runtime.

  • A complete re-write of layout and whitespace handling rules (see Let’s talk about whitespace), and with that a change in how layout is configured (see Configuring Layout) and the combination of some rules that were previously separate. One example of this is that the legacy rules L001, L005, L006, L008, L023, L024, L039, L048 & L071 have been combined simply into LT01.

Example 2.0 config

To illustrate the points above, this is an illustrative example config for a 2.0 compatible project. Note that the config is fairly brief and sets only the values which differ from the default config.

[sqlfluff]
dialect = snowflake
templater = dbt
max_line_length = 120

# Exclude some specific rules based on a mixture of codes and names
exclude_rules = RF02, RF03, RF04, ST06, ST07, AM05, AM06, convention.left_join, layout.select_targets

[sqlfluff:indentation]
# Enabling implicit indents for this project.
# See https://docs.sqlfluff.com/en/stable/layout.html#configuring-indent-locations
allow_implicit_indents = True

# Add a few specific rule configurations, referenced by the rule names
# and not by the rule codes.
[sqlfluff:rules:capitalisation.keywords]
capitalisation_policy = lower

[sqlfluff:rules:capitalisation.identifiers]
capitalisation_policy = lower

[sqlfluff:rules:capitalisation.functions]
extended_capitalisation_policy = lower

# An example of setting a custom layout specification which
# is more lenient than default config.
[sqlfluff:layout:type:set_operator]
line_position = alone

Upgrading to 1.4

This release brings several internal changes, and acts as a prelude to 2.0.0. In particular, the following config values have changed:

  • sqlfluff:rules:L007:operator_new_lines has been changed to sqlfluff:layout:type:binary_operator:line_position.

  • sqlfluff:rules:comma_style and sqlfluff:rules:L019:comma_style have both been consolidated into sqlfluff:layout:type:comma:line_position.

If any of these values have been set in your config, they will be automatically translated to the new values at runtime, and a warning will be shown. To silence the warning, update your config file to the new values. For more details on configuring layout see Configuring Layout.

Upgrading to 1.3

This release brings several potentially breaking changes to the underlying parse tree. For users of the cli tool in a linting context you should notice no change. If however your application relies on the structure of the SQLFluff parse tree or the naming of certain elements within the yaml format, then this may not be a drop-in replacement. Specifically:

  • The addition of a new end_of_file` meta segment at the end of the parse structure.

  • The addition of a template_loop` meta segment to signify a jump backward in the source file within a loop structure (e.g. a jinja for` loop).

  • Much more specific types on some raw segments, in particular identifier and literal type segments will now appear in the parse tree with their more specific type (which used to be called name) e.g. naked_identifier, quoted_identifier, numeric_literal etc…

If using the python api, the parent type (such as identifier) will still register if you call .is_type("identifier"), as this function checks all inherited types. However the eventual type returned by .get_type()` will now be (in most cases) what used to be accessible at .name. The name attribute will be deprecated in a future release.

Upgrading to 1.2

This release introduces the capability to automatically skip large files, and sets default limits on the maximum file size before a file is skipped. Users should see a performance gain, but may experience warnings associated with these skipped files.

Upgrades pre 1.0

  • 0.13.x new rule for quoted literals, option to remove hanging indents in rule L003, and introduction of ignore_words_regex.

  • 0.12.x dialect is now mandatory, the spark3 dialect was renamed to sparksql and datatype capitalisation was extracted from L010 to it’s own rule L063.

  • 0.11.x rule L030 changed to use extended_capitalisation_policy.

  • 0.10.x removed support for older dbt versions < 0.20 and stopped fix attempting to fix unparsable SQL.

  • 0.9.x refinement of the Simple API, dbt 1.0.0 compatibility, and the official SQLFluff Docker image.

  • 0.8.x an improvement to the performance of the parser, a rebuild of the Jinja Templater, and a progress bar for the CLI.

  • 0.7.x extracted the dbt templater to a separate plugin and removed the exasol_fs dialect (now merged in with the main exasol).

  • 0.6.x introduced parallel processing, which necessitated a big re-write of several innards.

  • 0.5.x introduced some breaking changes to the API.

  • 0.4.x dropped python 3.5, added the dbt templater, source mapping and also introduced the python API.

  • 0.3.x drops support for python 2.7 and 3.4, and also reworks the handling of indentation linting in a potentially not backward compatible way.

  • 0.2.x added templating support and a big restructure of rules and changed how users might interact with SQLFluff on templated code.

  • 0.1.x involved a major re-write of the parser, completely changing the behaviour of the tool with respect to complex parsing.