This page aims to act as a guide for migrating between major SQLFluff releases. Necessarily this means that bugfix releases, or releases requiring no change for the user are not mentioned. For full details of each individual release, see the detailed changelog.
Upgrading to 2.2¶
This release changes some of the interfaces between SQLFluff core and our plugin ecosystem. The only breaking change is in the interface between SQLFluff and templater plugins (which are not common in the ecosystem, hence why this is only a minor and not a major release).
For all plugins, we also recommend a different structure for their imports (especially for rule plugins which are more common in the ecosystem) - for performance and stability reasons. Some users had been experiencing very long import times with previous releases as a result of the layout of plugin imports. Users with affected plugins will begin to see a warning from this release onward, which can be resolved for their plugin by updating to a new version of that plugin which follows the guidelines.
Templaters before this version would pass a
callable to the slicing methods as part of being able to map the source
file. This method would accept a
str and return a
jinja2.environment.Template object to allow the templater to
render multiple variants of the template to do the slicing operation
(which allows linting issues found in templated files to be mapped
accurately back to their position in the unrendered source file).
This approach is not very generalisable, and did not support templating
operations with libraries other than
As a result, we have amended the interface to instead pass a
render_func() callable, which accepts a
str and returns
str. This works fine for the
jinja templater (and
by extension the
dbt templater) as they can simply wrap the
original callable with a method that calls
render() on the
Template object. It also however opens up the door
to other templating engines, and in particular to remote templaters
which might pass unrendered code over a HTTP connection for rendering.
slice_file()method of the base templater classes no longer accepts an optional
make_templateargument or a
render_funccallable should be passed which can be called to generate the
Unlike the optional
render_funcis not optional and should always be present.
We recommend that the module in a plugin which defines all
of the hook implementations (anything using the
must be able to fully import before any rule implementations are imported.
More specifically, SQLFluff must be able to both import and
run any implementations of
get_configs_info() before any plugin
rules (i.e. any derivatives of
BaseRule) are imported.
Because of this, we recommend that rules are defined in a
separate module to the root of the plugin and then only imported within
Importing in the main body of the module was previously our recommendation and so may be the case for versions of some plugins. If one of your plugins does use imports in this way, a warning will be presented from this version onward, recommending that you update your plugin.
See the Developing Plugins section of the docs for an example.
Upgrading from 1.x to 2.0¶
Upgrading to 2.0 brings several important breaking changes:
All bundled rules have been recoded, both from generic
L00Xformats into groups within similar codes (e.g. an aliasing group with codes of the format
AL0X), but also given names to allow much clearer referencing (e.g.
Rule Configuration now uses the rule name rather than the rule code to specify the section. Any unrecognised references in config files (whether they are references which do match existing rules by code or alias, or whether the match no rules at all) will raise warnings at runtime.
A complete re-write of layout and whitespace handling rules (see Let’s talk about whitespace), and with that a change in how layout is configured (see Configuring Layout) and the combination of some rules that were previously separate. One example of this is that the legacy rules
L071have been combined simply into
Recommended upgrade steps¶
To upgrade smoothly between versions, we recommend the following sequence:
The upgrade path will be simpler if you have a slimmer configuration file. Before upgrading, consider removing any sections from your configuration file (often
.sqlfluff, see Configuration) which match the current Default Configuration. There is no need to respecify defaults in your local config if they are not different to the stock config.
In a local (or other non-production) environment, upgrade to SQLFluff 2.0.x. We recommend using a compatible release specifier such as
~=2.0.0, to ensure any minor bugfix releases are automatically included.
Examine your configuration file (as mentioned above), and evaluate how rules are currently specified. We recommend primarily using either
exclude_rulesrather than both, as detailed in Enabling and Disabling Rules. Using either the
sqlfluff rulesCLI command or the online Rules Reference, replace all references to legacy rule codes (i.e. codes of the form
exclude_rulesconfig values. Here, consider using group specifiers or names to make your config simpler to read and understand (e.g.
capitalisation, is much more understandable than
CP01,CP02,CP03,CP04,CP05, but the two specifiers will have the same effect). Note that while legacy codes will still be understood here (because they remain valid as aliases for those rules) - you may find that some rules no longer exist in isolation and so these references may be misleading. e.g.
L005is now an alias for
layout.spacingbut that rule is much more broad ranging than the original scope of
L005, which was only spacing around commas.
In Rule Configuration. In particular here, legacy references to rule codes are no longer valid, will raise warnings, and until resolved, the configuration in those sections will be ignored. The new section references should include the rule name (e.g.
[sqlfluff:rules:L010]). This switch is designed to make configuration files more readable, but we cannot support backward compatibility here without also having to resolve the potential ambiguity of the scenario where both code-based and name-based are both used.
Review the Configuring Layout documentation, and check whether any indentation or layout configuration should be revised.
Check your project for In-File Configuration Directives which refer to rule codes. Alter these in the same manner as described above for configuration files.
Test linting your project for unexpected linting issues. Where found, consider whether to use
sqlfluff fixto repair them in bulk, or (if you disagree with the changes) consider changing which rules you enable or their configuration accordingly. In particular you may notice:
The indentation rule (
L003as was, now
LT02) has had a significant rewrite, and while much more flexible and accurate, it is also more specific. Note that Hanging Indents are no longer supported, and that while not enabled by default, many users may find the enabling Implicit Indents fits their organisation’s style better.
The spacing rule (
layout.spacing) has a much wider scope, and so may pick up spacing issues that were not previously enforced. If you disagree with any of these, you can override the
sqlfluff:layoutsections of the config with different (or just more liberal settings, like
Example 2.0 config¶
To illustrate the points above, this is an illustrative example config for a 2.0 compatible project. Note that the config is fairly brief and sets only the values which differ from the default config.
[sqlfluff] dialect = snowflake templater = dbt max_line_length = 120 # Exclude some specific rules based on a mixture of codes and names exclude_rules = RF02, RF03, RF04, ST06, ST07, AM05, AM06, convention.left_join, layout.select_targets [sqlfluff:indentation] # Enabling implicit indents for this project. # See https://docs.sqlfluff.com/en/stable/layout.html#configuring-indent-locations allow_implicit_indents = True # Add a few specific rule configurations, referenced by the rule names # and not by the rule codes. [sqlfluff:rules:capitalisation.keywords] capitalisation_policy = lower [sqlfluff:rules:capitalisation.identifiers] capitalisation_policy = lower [sqlfluff:rules:capitalisation.functions] extended_capitalisation_policy = lower # An example of setting a custom layout specification which # is more lenient than default config. [sqlfluff:layout:type:set_operator] line_position = alone
Upgrading to 1.4¶
This release brings several internal changes, and acts as a prelude to 2.0.0. In particular, the following config values have changed:
sqlfluff:rules:L007:operator_new_lineshas been changed to
sqlfluff:rules:L019:comma_stylehave both been consolidated into
If any of these values have been set in your config, they will be automatically translated to the new values at runtime, and a warning will be shown. To silence the warning, update your config file to the new values. For more details on configuring layout see Configuring Layout.
Upgrading to 1.3¶
This release brings several potentially breaking changes to the underlying parse tree. For users of the cli tool in a linting context you should notice no change. If however your application relies on the structure of the SQLFluff parse tree or the naming of certain elements within the yaml format, then this may not be a drop-in replacement. Specifically:
The addition of a new
end_of_file`meta segment at the end of the parse structure.
The addition of a
template_loop`meta segment to signify a jump backward in the source file within a loop structure (e.g. a jinja
Much more specific types on some raw segments, in particular
literaltype segments will now appear in the parse tree with their more specific type (which used to be called
If using the python api, the parent type (such as
will still register if you call
.is_type("identifier"), as this
function checks all inherited types. However the eventual type returned
.get_type()` will now be (in most cases) what used to be
name attribute will be deprecated
in a future release.
Upgrading to 1.2¶
This release introduces the capability to automatically skip large files, and sets default limits on the maximum file size before a file is skipped. Users should see a performance gain, but may experience warnings associated with these skipped files.
Upgrades pre 1.0¶
0.13.x new rule for quoted literals, option to remove hanging indents in rule L003, and introduction of
0.12.x dialect is now mandatory, the
spark3dialect was renamed to
sparksqland datatype capitalisation was extracted from L010 to it’s own rule L063.
0.11.x rule L030 changed to use
0.10.x removed support for older dbt versions < 0.20 and stopped
fixattempting to fix unparsable SQL.
0.9.x refinement of the Simple API, dbt 1.0.0 compatibility, and the official SQLFluff Docker image.
0.8.x an improvement to the performance of the parser, a rebuild of the Jinja Templater, and a progress bar for the CLI.
0.7.x extracted the dbt templater to a separate plugin and removed the
exasol_fsdialect (now merged in with the main
0.6.x introduced parallel processing, which necessitated a big re-write of several innards.
0.5.x introduced some breaking changes to the API.
0.4.x dropped python 3.5, added the dbt templater, source mapping and also introduced the python API.
0.3.x drops support for python 2.7 and 3.4, and also reworks the handling of indentation linting in a potentially not backward compatible way.
0.2.x added templating support and a big restructure of rules and changed how users might interact with SQLFluff on templated code.
0.1.x involved a major re-write of the parser, completely changing the behaviour of the tool with respect to complex parsing.