Configuration

SQLFluff accepts configuration either through the command line or through configuration files. There is rough parity between the two approaches with the exception that templating configuration must be done via a file, because it otherwise gets slightly complicated.

For details of what’s available on the command line check out the CLI Reference.

Configuration Files

For file based configuration SQLFluff will look for the following files in order. Later files will (if found) will be used to overwrite any values read from earlier files.

  • setup.cfg

  • tox.ini

  • pep8.ini

  • .sqlfluff

  • pyproject.toml

Within these files, the first four will be read like a cfg file, and SQLFluff will look for sections which start with sqlfluff, and where subsections are delimited by a semicolon. For example the jinjacontext section will be indicated in the section started with [sqlfluff:jinjacontext].

For example, a snippet from a .sqlfluff file (as well as any of the supported cfg file types):

[sqlfluff]
templater = jinja
sql_file_exts = .sql,.sql.j2,.dml,.ddl

[sqlfluff:indentation]
indented_joins = False
indented_using_on = True
template_blocks_indent = False

[sqlfluff:templater]
unwrap_wrapped_queries = True

[sqlfluff:templater:jinja]
apply_dbt_builtins = True

For the pyproject.toml file, all valid sections start with tool.sqlfluff and subsections are delimited by a dot. For example the jinjacontext section will be indicated in the section started with [tool.sqlfluff.jinjacontext].

For example, a snippet from a pyproject.toml file:

[tool.sqlfluff.core]
templater = "jinja"
sql_file_exts = ".sql,.sql.j2,.dml,.ddl"

[tool.sqlfluff.indentation]
indented_joins = false
indented_using_on = true
template_blocks_indent = false

[tool.sqlfluff.templater]
unwrap_wrapped_queries = true

[tool.sqlfluff.templater.jinja]
apply_dbt_builtins = true

# For rule specific configuration, use dots between the names exactly
# as you would in .sqlfluff. In the background, SQLFluff will unpack the
# configuration paths accordingly.
[tool.sqlfluff.rules.capitalisation.keywords]
capitalisation_policy = "upper"

New Project Configuration

When setting up a new project with SQLFluff, we recommend keeping your configuration file fairly minimal. The config file should act as a form of documentation for your team i.e. a record of what decisions you’ve made which govern how your format your SQL. By having a more concise config file, and only defining config settings where they differ from the defaults - you are more clearly stating to your team what choices you’ve made.

However, there are also a few places where the default configuration is designed more for existing projects, rather than fresh projects, and so there is an opportunity to be a little stricter than you might otherwise be with an existing codebase.

Here is a simple configuration file which would be suitable for a starter project:

[sqlfluff]

# Supported dialects https://docs.sqlfluff.com/en/stable/dialects.html
# Or run 'sqlfluff dialects'
dialect = snowflake

# One of [raw|jinja|python|placeholder]
templater = jinja

# Comma separated list of rules to exclude, or None
# See https://docs.sqlfluff.com/en/stable/configuration.html#enabling-and-disabling-rules
# AM04 (ambiguous.column_count) and ST06 (structure.column_order) are
# two of the more controversial rules included to illustrate usage.
exclude_rules = ambiguous.column_count, structure.column_order

# The standard max_line_length is 80 in line with the convention of
# other tools and several style guides. Many projects however prefer
# something a little longer.
# Set to zero or negative to disable checks.
max_line_length = 120

# CPU processes to use while linting.
# The default is "single threaded" to allow easy debugging, but this
# is often undesirable at scale.
# If positive, just implies number of processes.
# If negative or zero, implies number_of_cpus - specified_number.
# e.g. -1 means use all processors but one. 0 means all cpus.
processes = -1

# If using the dbt templater, we recommend setting the project dir.
[sqlfluff:templater:dbt]
project_dir = ./

[sqlfluff:indentation]
# While implicit indents are not enabled by default. Many of the
# SQLFluff maintainers do use them in their projects.
allow_implicit_indents = True

[sqlfluff:rules:aliasing.length]
min_alias_length = 3

# The default configuration for capitalisation rules is "consistent"
# which will auto-detect the setting from the rest of the file. This
# is less desirable in a new project and you may find this (slightly
# more strict) setting more useful.
# Typically we find users rely on syntax highlighting rather than
# capitalisation to distinguish between keywords and identifiers.
# Clearly, if your organisation has already settled on uppercase
# formatting for any of these syntax elements then set them to "upper".
# See https://stackoverflow.com/questions/608196/why-should-i-capitalize-my-sql-keywords-is-there-a-good-reason
[sqlfluff:rules:capitalisation.keywords]
capitalisation_policy = lower
[sqlfluff:rules:capitalisation.identifiers]
extended_capitalisation_policy = lower
[sqlfluff:rules:capitalisation.functions]
extended_capitalisation_policy = lower
[sqlfluff:rules:capitalisation.literals]
capitalisation_policy = lower
[sqlfluff:rules:capitalisation.types]
extended_capitalisation_policy = lower

# The default configuration for the not equal convention rule is "consistent"
# which will auto-detect the setting from the rest of the file. This
# is less desirable in a new project and you may find this (slightly
# more strict) setting more useful.
[sqlfluff:rules:convention.not_equal]
# Default to preferring the "c_style" (i.e. `!=`)
preferred_not_equal_style = c_style

Nesting

SQLFluff uses nesting in its configuration files, with files closer overriding (or patching, if you will) values from other files. That means you’ll end up with a final config which will be a patchwork of all the values from the config files loaded up to that path. The exception to this is the value for templater, which cannot be set in config files in subdirectories of the working directory. You don’t need any config files to be present to make SQLFluff work. If you do want to override any values though SQLFluff will use files in the following locations in order, with values from later steps overriding those from earlier:

  1. […and this one doesn’t really count] There’s a default config as part of the SQLFluff package. You can find this below, in the Default Configuration section.

  2. It will look in the user’s os-specific app config directory. On macOS and Unix this is ~/.config/sqlfluff, Windows is <home>\AppData\Local\sqlfluff\sqlfluff, for any of the filenames above in the main Configuration section. If multiple are present, they will patch/override each other in the order above.

  3. It will look for the same files in the user’s home directory (~).

  4. It will look for the same files in the current working directory.

  5. [if parsing a file in a subdirectory of the current working directory] It will look for the same files in every subdirectory between the current working dir and the file directory.

  6. It will look for the same files in the directory containing the file being linted.

This whole structure leads to efficient configuration, in particular in projects which utilise a lot of complicated templating.

In-File Configuration Directives

In addition to configuration files mentioned above, SQLFluff also supports comment based configuration switching in files. This allows specific SQL file to modify a default configuration if they have specific needs.

When used, these apply to the whole file, and are parsed from the file in an initial step before the rest of the file is properly parsed. This means they can be used for both rule configuration and also for parsing configuration.

To use these, the syntax must start as an inline sql comment beginning with sqlfluff (i.e. -- sqlfluff). The line is then interpreted as a colon-seperated address of the configuation value you wish to set. A few common examples are shown below:

-- Set Indented Joins
-- sqlfluff:indentation:indented_joins:True

-- Set a smaller indent for this file
-- sqlfluff:indentation:tab_space_size:2

-- Set keywords to be capitalised
-- sqlfluff:rules:capitalisation.keywords:capitalisation_policy:upper

SELECT *
FROM a
  JOIN b USING(c)

We recommend only using this configuration approach for configuration that applies to one file in isolation. For configuration changes for areas of a project or for whole projects we recommend Nesting of configuration files.

Rule Configuration

Rules can be configured with the .sqlfluff config files.

Common rule configurations can be set in the [sqlfluff:rules] section.

For example:

[sqlfluff:rules]
allow_scalar = True
single_table_references = consistent
unquoted_identifiers_policy = all

Rule specific configurations are set in rule specific subsections.

For example, enforce that keywords are upper case by configuring the rule CP01:

[sqlfluff:rules:capitalisation.keywords]
# Keywords
capitalisation_policy = upper

All possible options for rule sections are documented in Rules Reference.

For an overview of the most common rule configurations that you may want to tweak, see Default Configuration (and use Rules Reference to find the available alternatives).

Enabling and Disabling Rules

The decision as to which rules are applied to a given file is applied on a file by file basis, by the effective configuration for that file. There are two configuration values which you can use to set this:

  • rules, which explicitly enables the specified rules. If this parameter is unset or empty for a file, this implies “no selection” and so “all rules” is taken to be the meaning.

  • exclude_rules, which explicitly disables the specified rules. This parameter is applied after the rules parameter so can be used to subtract from the otherwise enabled set.

Each of these two configuration values accept a comma separated list of references. Each of those references can be:

  • a rule code e.g. LN01

  • a rule name e.g. layout.indent

  • a rule alias, which is often a deprecated code e.g. L003

  • a rule group e.g. layout or capitalisation

These different references can be mixed within a given expression, which results in a very powerful syntax for selecting exactly which rules are active for a given file.

Note

It’s worth mentioning here that the application of rules and exclude_rules, with groups, aliases and names, in projects with potentially multiple nested configuration files defining different rules for different areas of a project can get very confusing very fast. While this flexibility is intended for users to take advantage of, we do have some recommendations about how to do this is a way that remains manageable.

When considering configuration inheritance, each of rules and exclude_rules will totally overwrite any values in parent config files if they are set in a child file. While the subtraction operation between both of them is calculated “per file”, there is no combination operation between two definitions of rules (just one overwrites the other).

The effect of this is that we recommend one of two approaches:

  1. Simply only use rules. This has the upshot of each area of your project being very explicit in which rules are enabled. When that changes for part of your project you just reset the whole list of applicable rules for that part of the project.

  2. Set a single rules value in your master project config file and then only use exclude_rules in sub-configuration files to turn off specific rules for parts of the project where those rules are inappropriate. This keeps the simplicity of only having one value which is inherited, but allows slightly easier and simpler rollout of new rules because we manage by exception.

For example, to disable the rules LT08 and RF02:

[sqlfluff]
exclude_rules = LT08, RF02

To enable individual rules, configure rules, respectively.

For example, to enable RF02:

[sqlfluff]
rules = RF02

Rules can also be enabled/disabled by their grouping. Right now, the only rule grouping is core. This will enable (or disable) a select group of rules that have been deemed ‘core rules’.

[sqlfluff]
rules = core

More information about ‘core rules’ can be found in the Rules Reference.

Additionally, some rules have a special force_enable configuration option, which allows to enable the given rule even for dialects where it is disabled by default. The rules that support this can be found in the Rules Reference.

The default values can be seen in Default Configuration.

See also: Ignoring Errors & Files.

Downgrading rules to warnings

To keep displaying violations for specific rules, but not have those issues lead to a failed run, rules can be downgraded to warnings. Rules set as warnings won’t cause a file to fail, but will still be shown in the CLI to warn users of their presence.

The configuration of this behaves very like exclude_rules above:

[sqlfluff]
warnings = LT01, LT04

With this configuration, files with no other issues (other than those set to warn) will pass. If there are still other issues, then the file will still fail, but will show both warnings and failures.

== [test.sql] PASS
L:   2 | P:   9 | LT01 | WARNING: Missing whitespace before +
== [test2.sql] FAIL
L:   2 | P:   8 | CP02 | Unquoted identifiers must be consistently upper case.
L:   2 | P:  11 | LT01 | WARNING: Missing whitespace before +

This is particularly useful as a transitional tool when considering the introduction on new rules on a project where you might want to make users aware of issues without blocking their workflow (yet).

Layout & Spacing Configuration

The [sqlfluff:layout] section of the config controls the treatment of spacing and line breaks across all rules. To understand more about this section, see the section of the docs dedicated to layout: Configuring Layout.

Templating Configuration

This section explains how to configure templating for SQL files.

When writing SQL files, users might utilise some kind of templating. The SQL file itself is written with placeholders which get rendered to proper SQL at run time. This can range from very simple placeholder templating to complex Jinja templating.

SQLFluff supports templated sections in SQL, see Stage 1, the templater. This is achieved by the following set of operations:

  1. SQLFluff pre-renders the templated SQL

  2. SQLFluff applies the lint and fix operations to the rendered file

  3. SQLFluff backports the rule violations to the templated section of the SQL.

SQLFluff does not automatically have access to the same environment used in production template setup. This means it is necessary to either provide that environment or provide dummy values to effectively render the template and generate valid SQL. Refer to the templater sections below for details.

SQLFluff natively supports the following templating engines

Also, SQLFluff has an integration to use dbt as a templater.

  • dbt templater (via plugin which is covered in a different section).

Note

Templaters may not be able to generate a rendered SQL that cover the entire raw file.

For example, if the raw SQL uses a {% if condition %} block, the rendered version of the template will only include either the {% then %} or the {% else %} block (depending on the provided configuration for the templater), but not both.

In this case, because SQLFluff linting can only operate on the output of the templater, some areas of the raw SQL will never be seen by the linter and will not be covered by lint rules.

This is functionality we hope to support in future.

Generic Variable Templating

Variables are available in all the templaters. By default the templating engine will expect variables for templating to be available in the config, and the templater will be look in the section corresponding to the context for that templater. By convention, the config for the jinja templater is found in the sqlfluff:templater:jinja:context section, the config for the python templater is found in the sqlfluff:templater:python:context section, the one for the placeholder templater is found in the sqlfluff:templater:placeholder:context section.

For example, if passed the following .sql file:

SELECT {{ num_things }} FROM {{ tbl_name }} WHERE id > 10 LIMIT 5

…and the following configuration in .sqlfluff in the same directory:

[sqlfluff:templater:jinja:context]
num_things=456
tbl_name=my_table

…then before parsing, the sql will be transformed to:

SELECT 456 FROM my_table WHERE id > 10 LIMIT 5

Note

If there are variables in the template which cannot be found in the current configuration context, then this will raise a SQLTemplatingError and this will appear as a violation without a line number, quoting the name of the variable that couldn’t be found.

Jinja templater

The Jinja templater uses Jinja2 to render templates.

There are multiple, complementary ways of configuring the Jinja templater.

  • Reading variables and Jinja macros directly from the SQLFLuff config file

  • Loading macros from a path

  • Using a library

Overview of Jinja templater’s configuration options

Configuration

Variables

Macros

Filters

Documentation

Config file

Complex Jinja Variable Templating and Jinja Macro Templating (from config)

Macro Path

Jinja Macro Templating (from file)

Library

Library Templating

For example, a snippet from a .sqlfluff file that uses all config options:

[sqlfluff]
templater = jinja

[sqlfluff:templater:jinja]
apply_dbt_builtins = True
load_macros_from_path = my_macros
library_path = sqlfluff_libs

[sqlfluff:templater:jinja:context]
my_list = ['a', 'b', 'c']
MY_LIST = ("d", "e", "f")
my_where_dict = {"field_1": 1, "field_2": 2}

[sqlfluff:templater:jinja:macros]
a_macro_def = {% macro my_macro(n) %}{{ n }} + {{ n * 2 }}{% endmacro %}

Complex Jinja Variable Templating

Apart from the Generic variable templating that is supported for all templaters, two more advanced features of variable templating are available for Jinja.

case sensitivity and native python types. Both are illustrated in the following example:

[sqlfluff:templater:jinja:context]
my_list = ['a', 'b', 'c']
MY_LIST = ("d", "e", "f")
my_where_dict = {"field_1": 1, "field_2": 2}
SELECT
    {% for elem in MY_LIST %}
        '{{elem}}' {% if not loop.last %}||{% endif %}
    {% endfor %} as concatenated_list
FROM tbl
WHERE
    {% for field, value in my_where_dict.items() %}
        {{field}} = {{value}} {% if not loop.last %}and{% endif %}
    {% endfor %}

…will render as…

SELECT
    'd' || 'e' || 'f' as concatenated_list
FROM tbl
WHERE
    field_1 = 1 and field_2 = 2

Note that the variable was replaced in a case sensitive way and that the settings in the config file were interpreted as native python types.

Jinja Macro Templating (from config)

Macros (which also look and feel like functions are available only in the jinja templater. Similar to Generic Variable Templating, these are specified in config files, what’s different in this case is how they are named. Similar to the context section above, macros are configured separately in the macros section of the config. Consider the following example.

If passed the following .sql file:

SELECT {{ my_macro(6) }} FROM some_table

…and the following configuration in .sqlfluff in the same directory (note the tight control of whitespace):

[sqlfluff:templater:jinja:macros]
a_macro_def = {% macro my_macro(n) %}{{ n }} + {{ n * 2 }}{% endmacro %}

…then before parsing, the sql will be transformed to:

SELECT 6 + 12 FROM some_table

Note that in the code block above, the variable name in the config is a_macro_def, and this isn’t apparently otherwise used anywhere else. Broadly this is accurate, however within the configuration loader this will still be used to overwrite previous values in other config files. As such this introduces the idea of config blocks which could be selectively overwritten by other configuration files downstream as required.

Jinja Macro Templating (from file)

In addition to macros specified in the config file, macros can also be loaded from files or folders. This is specified in the config file:

[sqlfluff:templater:jinja]
load_macros_from_path = my_macros

load_macros_from_path is a comma-separated list of .sql files or folders. Locations are relative to the config file. For example, if the config file above was found at /home/my_project/.sqlfluff, then SQLFluff will look for macros in the folder /home/my_project/my_macros/ (but not subfolders). Any macros defined in the config will always take precedence over a macro defined in the path.

  • .sql files: Macros in these files are available in every .sql file without requiring a Jinja include or import.

  • Folders: To use macros from the .sql files in folders, use Jinja include or import as explained below.

Note: The load_macros_from_path setting also defines the search path for Jinja include or import. Unlike with macros (as noted above), subdirectories are supported. For example, if load_macros_from_path is set to my_macros, and there is a file my_macros/subdir/my_file.sql, you can do:

{% include 'subdir/include_comment.sql' %}

Note

Throughout the templating process whitespace will still be treated rigorously, and this includes newlines. In particular you may choose to provide dummy macros in your configuration different from the actual macros used in production.

REMEMBER: The reason SQLFluff supports macros is to enable it to parse templated sql without it being a blocker. It shouldn’t be a requirement that the templating is accurate - it only needs to work well enough that parsing and linting are helpful.

Builtin Jinja Macro Blocks

One of the main use cases which inspired SQLFluff as a project was dbt. It uses jinja templating extensively and leads to some users maintaining large repositories of sql files which could potentially benefit from some linting.

Note

SQLFluff has now a tighter integration with dbt through the “dbt” templater. It is the recommended templater for dbt projects. If used, it eliminates the need for the overrides described in this section.

To use the dbt templater, go to dbt templater.

SQLFluff anticipates this use case and provides some built in macro blocks in the Default Configuration which assist in getting started with dbt projects. In particular it provides mock objects for:

  • ref: The mock version of this provided simply returns the model reference as the name of the table. In most cases this is sufficient.

  • config: A regularly used macro in dbt to set configuration values. For linting purposes, this makes no difference and so the provided macro simply returns nothing.

Note

If there are other builtin macros which would make your life easier, consider submitting the idea (or even better a pull request) on github.

Library Templating

If using SQLFluff with jinja as your templater, you may have library function calls within your sql files that can not be templated via the normal macro templating mechanisms:

SELECT foo, bar FROM baz {{ dbt_utils.group_by(2) }}

To template these libraries, you can use the sqlfluff:jinja:library_path config option:

[sqlfluff:templater:jinja]
library_path = sqlfluff_libs

This will pull in any python modules from that directory and allow sqlfluff to use them in templates. In the above example, you might define a file at sqlfluff_libs/dbt_utils.py as:

def group_by(n):
    return "GROUP BY 1,2"

If an __init__.py is detected, it will be loaded alongside any modules and submodules found within the library path.

SELECT
   {{ custom_sum('foo', 'bar') }},
   {{ foo.bar.another_sum('foo', 'bar') }}
FROM
   baz

sqlfluff_libs/__init__.py:

def custom_sum(a: str, b: str) -> str:
    return a + b

sqlfluff_libs/foo/__init__.py:

# empty file

sqlfluff_libs/foo/bar.py:

def another_sum(a: str, b: str) -> str:
   return a + b

Additionally, the library can be used to expose Jinja Filters to the Jinja environment used by SQLFluff.

This is achieve by setting a global variable named SQLFLUFF_JINJA_FILTERS. SQLFLUFF_JINJA_FILTERS is a dictionary where

  • dictionary keys map to the Jinja filter name

  • dictionary values map to the Python callable

For example, to make the Airflow filter ds available to SQLFLuff, add the following to the __init__.py of the library:

# https://github.com/apache/airflow/blob/main/airflow/templates.py#L53
def ds_filter(value: datetime.date | datetime.time | None) -> str | None:
   """Date filter."""
   if value is None:
       return None
   return value.strftime("%Y-%m-%d")

SQLFLUFF_JINJA_FILTERS = {"ds": ds_filter}

Now, ds can be used in SQL

SELECT "{{ "2000-01-01" | ds }}";

Interaction with --ignore=templating

Ignoring Jinja templating errors provides a way for users to use SQLFluff while reducing or avoiding the need to spend a lot of time adding variables to [sqlfluff:templater:jinja:context].

When --ignore=templating is enabled, the Jinja templater behaves a bit differently. This additional behavior is usually but not always helpful for making the file at least partially parsable and fixable. It definitely doesn’t guarantee that every file can be fixed, but it’s proven useful for some users.

Here’s how it works:

  • Within the expanded SQL, undefined variables are automatically replaced with the corresponding string value.

  • If you do: {% include query %}, and the variable query is not defined, it returns a “file” containing the string query.

  • If you do: {% include "query_file.sql" %}, and that file does not exist or you haven’t configured a setting for load_macros_from_path, it returns a “file” containing the text query_file.

For example:

select {{ my_variable }}
from {% include "my_table.sql" %}

is interpreted as:

select my_variable
from my_table

The values provided by the Jinja templater act a bit (not exactly) like a mixture of several types:

  • str

  • int

  • list

  • Jinja’s Undefined class

Because the values behave like Undefined, it’s possible to replace them using Jinja’s default() filter. For example:

select {{ my_variable | default("col_a") }}
from my_table

is interpreted as:

select col_a
from my_table

Placeholder templater

Libraries such as SQLAlchemy or Psycopg use different parameter placeholder styles to mark where a parameter has to be inserted in the query.

For example a query in SQLAlchemy can look like this:

SELECT * FROM table WHERE id = :myid

At runtime :myid will be replace by a value provided by the application and escaped as needed, but this is not standard SQL and cannot be parsed as is.

In order to parse these queries is then necessary to replace these placeholders with sample values, and this is done with the placeholder templater.

Placeholder templating can be enabled in the config using:

[sqlfluff]
templater = placeholder

A few common styles are supported:

 -- colon
 WHERE bla = :my_name

 -- colon_nospaces
 -- (use with caution as more prone to false positives)
 WHERE bla = table:my_name

 -- numeric_colon
 WHERE bla = :2

 -- pyformat
 WHERE bla = %(my_name)s

 -- dollar
 WHERE bla = $my_name or WHERE bla = ${my_name}

 -- question_mark
 WHERE bla = ?

 -- numeric_dollar
 WHERE bla = $3 or WHERE bla = ${3}

 -- percent
 WHERE bla = %s

 -- ampersand
 WHERE bla = &s or WHERE bla = &{s} or USE DATABASE MARK_{ENV}

These can be configured by setting param_style to the names above:

[sqlfluff:templater:placeholder]
param_style = colon
my_name = 'john'

then you can set sample values for each parameter, like my_name above. Notice that the value needs to be escaped as it will be replaced as a string during parsing. When the sample values aren’t provided, the templater will use parameter names themselves by default.

When parameters are positional, like question_mark, then their name is simply the order in which they appear, starting with 1.

[sqlfluff:templater:placeholder]
param_style = question_mark
1 = 'john'

In case you need a parameter style different from the ones above, you can pass a custom regex.

[sqlfluff:templater:placeholder]
param_regex = __(?P<param_name>[\w_]+)__
my_name = 'john'

N.B. quotes around param_regex in the config are interpreted literally by the templater. e.g. param_regex=’__(?P<param_name>[w_]+)__’ matches ‘__some_param__’ not __some_param__

the named parameter param_name will be used as the key to replace, if missing, the parameter is assumed to be positional and numbers are used instead.

Also consider making a pull request to the project to have your style added, it may be useful to other people and simplify your configuration.

Python templater

Uses native Python f-strings. As described in Generic Variable Templating, an example usage would look be configured as follows:

If passed the following .sql file:

SELECT * FROM {tbl_name}

…and the following configuration in .sqlfluff in the same directory:

[sqlfluff]
templater = python

[sqlfluff:templater:python:context]
tbl_name = my_table

…then before parsing, the sql will be transformed to:

SELECT * FROM my_table

dbt templater

Note

From sqlfluff version 0.7.0 onwards, the dbt templater has been moved to a separate plugin and python package. Projects that were already using the dbt templater may initially fail after an upgrade to 0.7.0+. See the installation instructions below to install the dbt templater.

dbt templating is still a relatively new feature added in 0.4.0 and is still in very active development! If you encounter an issue, please let us know in a GitHub issue or on the SQLFluff slack workspace.

dbt is not the default templater for SQLFluff (it is jinja). dbt is a complex tool, so using the default jinja templater will be simpler. You should be aware when using the dbt templater that you will be exposed to some of the complexity of dbt. Users may wish to try both templaters and choose according to how they intend to use SQLFluff.

A simple rule of thumb might be:

  • If you are using SQLFluff in a CI/CD context, where speed is not critical but accuracy in rendering sql is, then the dbt templater may be more appropriate.

  • If you are using SQLFluff in an IDE or on a git hook, where speed of response may be more important, then the jinja templater may be more appropriate.

Pros:

  • Most (potentially all) macros will work

Cons:

  • More complex, e.g. using it successfully may require deeper understanding of your models and/or macros (including third-party macros)

    • More configuration decisions to make

    • Best practices are not yet established or documented

  • If your dbt model files access a database at compile time, using SQLFluff with the dbt templater will also require access to a database.

    • Note that you can often point SQLFluff and the dbt templater at a test database (i.e. it doesn’t have to be the production database).

  • Runs slower

Installation & Configuration

In order to get started using SQLFluff with a dbt project you will first need to install the relevant dbt adapter for your dialect and the sqlfluff-templater-dbt package using your package manager of choice (e.g. pip install dbt-postgres sqlfluff-templater-dbt) and then will need the following configuration:

In .sqlfluff:

[sqlfluff]
templater = dbt

In .sqlfluffignore:

target/
# dbt <1.0.0
dbt_modules/
# dbt >=1.0.0
dbt_packages/
macros/

You can set the dbt project directory, profiles directory and profile with:

[sqlfluff:templater:dbt]
project_dir = <relative or absolute path to dbt_project directory>
profiles_dir = <relative or absolute path to the directory that contains the profiles.yml file>
profile = <dbt profile>
target = <dbt target>

Note

If the profiles_dir setting is omitted, SQLFluff will look for the profile in the default location, which varies by operating system. On Unix-like operating systems (e.g. Linux or macOS), the default profile directory is ~/.dbt/. On Windows, you can determine your default profile directory by running dbt debug –config-dir.

To use builtin dbt Jinja functions SQLFluff provides a configuration option that enables usage within templates.

[sqlfluff:templater:jinja]
apply_dbt_builtins = True

This will provide dbt macros like ref, var, is_incremental(). If the need arises builtin dbt macros can be customised via Jinja macros in .sqlfluff configuration file.

[sqlfluff:templater:jinja:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable, default='') %}item{% endmacro %}
dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}

If your project requires that you pass variables to dbt through command line, you can specify them in template:dbt:context section of .sqlfluff. See below configuration and its equivalent dbt command:

[sqlfluff:templater:dbt:context]
my_variable = 1
dbt run --vars '{"my_variable": 1}'

Known Caveats

  • To use the dbt templater, you must set templater = dbt in the .sqlfluff config file in the directory where sqlfluff is run. The templater cannot be changed in .sqlfluff files in subdirectories.

  • In SQLFluff 0.4.0 using the dbt templater requires that all files within the root and child directories of the dbt project must be part of the project. If there are deployment scripts which refer to SQL files not part of the project for instance, this will result in an error. You can overcome this by adding any non-dbt project SQL files to .sqlfluffignore.

CLI Arguments

You already know you can pass arguments (--verbose, --exclude-rules, etc.) through the CLI commands (lint, fix, etc.):

$ sqlfluff lint my_code.sql -v --exclude-rules LT08,RF02

You might have arguments that you pass through every time, e.g rules you always want to ignore. These can also be configured:

[sqlfluff]
verbose = 1
exclude_rules = LT08,RF02

Note that while the exclude_rules config looks similar to the above example, the verbose config has an integer value. This is because verbose is stackable meaning there are multiple levels of verbosity that are available for configuration. See CLI Reference for more details about the available CLI arguments. For more details about rule exclusion, see Enabling and Disabling Rules.

Ignoring Errors & Files

Ignoring individual lines

Similar to flake8’s ignore, individual lines can be ignored by adding -- noqa to the end of the line. Additionally, specific rules can be ignored by quoting their code or the category.

-- Ignore all errors
SeLeCt  1 from tBl ;    -- noqa

-- Ignore rule CP02 & rule CP03
SeLeCt  1 from tBl ;    -- noqa: CP02,CP03

-- Ignore all parsing errors
SeLeCt from tBl ;       -- noqa: PRS

Ignoring line ranges

Similar to pylint’s “pylint” directive”, ranges of lines can be ignored by adding -- noqa:disable=<rule>[,...] | all to the line. Following this directive, specified rules (or all rules, if “all” was specified) will be ignored until a corresponding – noqa:enable=<rule>[,…] | all directive.

-- Ignore rule AL02 from this line forward
SELECT col_a a FROM foo -- noqa: disable=AL02

-- Ignore all rules from this line forward
SELECT col_a a FROM foo -- noqa: disable=all

-- Enforce all rules from this line forward
SELECT col_a a FROM foo -- noqa: enable=all

Ignoring types of errors

General categories of errors can be ignored using the --ignore command line option or the ignore setting in .sqlfluffignore. Types of errors that can be ignored include:

  • lexing

  • linting

  • parsing

  • templating

.sqlfluffignore

Similar to Git’s .gitignore and Docker’s .dockerignore, SQLFluff supports a .sqlfluffignore file to control which files are and aren’t linted. Under the hood we use the python pathspec library which also has a brief tutorial in their documentation.

An example of a potential .sqlfluffignore placed in the root of your project would be:

# Comments start with a hash.

# Ignore anything in the "temp" path
/temp/

# Ignore anything called "testing.sql"
testing.sql

# Ignore any ".tsql" files
*.tsql

Ignore files can also be placed in subdirectories of a path which is being linted and the sub files will also be applied within that subdirectory.

Default Configuration

The default configuration is as follows, note the Builtin Jinja Macro Blocks in section [sqlfluff:templater:jinja:macros] as referred to above.

Note

This shows the entire default config. We do not recommend that users copy this whole config as the starter config file for their project.

This is for two reasons:

  1. The config file should act as a form of documentation for your team. A record of what decisions you’ve made which govern how your format your sql. By having a more concise config file, and only defining config settings where they differ from the defaults - you are more clearly stating to your team what choices you’ve made.

  2. As the project evolves, the structure of the config file may change and we will attempt to make changes as backward compatible as possible. If you have not overridden a config setting in your project, we can easily update the default config to match your expected behaviour over time. We may also find issues with the default config which we can also fix in the background. However, the longer your local config file, the more work it will be to update and migrate your config file between major versions.

If you are starting a fresh project and are looking for a good starter config, check out the New Project Configuration section above.

[sqlfluff]
# verbose is an integer (0-2) indicating the level of log output
verbose = 0
# Turn off color formatting of output
nocolor = False
# Supported dialects https://docs.sqlfluff.com/en/stable/dialects.html
# Or run 'sqlfluff dialects'
dialect = None
# One of [raw|jinja|python|placeholder]
templater = jinja
# Comma separated list of rules to check, default to all
rules = all
# Comma separated list of rules to exclude, or None
exclude_rules = None
# Below controls SQLFluff output, see max_line_length for SQL output
output_line_length = 80
# Number of passes to run before admitting defeat
runaway_limit = 10
# Ignore errors by category (one or more of the following, separated by commas: lexing,linting,parsing,templating)
ignore = None
# Warn only for rule codes (one of more rule codes, seperated by commas: e.g. LT01,LT02)
# Also works for templating and parsing errors by using TMP or PRS
warnings = None
# Whether to warn about unneeded '-- noqa:' comments.
warn_unused_ignores = False
# Ignore linting errors found within sections of code coming directly from
# templated code (e.g. from within Jinja curly braces. Note that it does not
# ignore errors from literal code found within template loops.
ignore_templated_areas = True
# can either be autodetect or a valid encoding e.g. utf-8, utf-8-sig
encoding = autodetect
# Ignore inline overrides (e.g. to test if still required)
disable_noqa = False
# Comma separated list of file extensions to lint
# NB: This config will only apply in the root folder
sql_file_exts = .sql,.sql.j2,.dml,.ddl
# Allow fix to run on files, even if they contain parsing errors
# Note altering this is NOT RECOMMENDED as can corrupt SQL
fix_even_unparsable = False
# Very large files can make the parser effectively hang.
# The more efficient check is the _byte_ limit check which
# is enabled by default. The previous _character_ limit check
# is still present for backward compatibility. This will be
# removed in a future version.
# Set either to 0 to disable.
large_file_skip_char_limit = 0
large_file_skip_byte_limit = 20000
# CPU processes to use while linting.
# If positive, just implies number of processes.
# If negative or zero, implies number_of_cpus - specified_number.
# e.g. -1 means use all processors but one. 0  means all cpus.
processes = 1
# Max line length is set by default to be in line with the dbt style guide.
# https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md
# Set to zero or negative to disable checks.
max_line_length = 80

[sqlfluff:indentation]
# See https://docs.sqlfluff.com/en/stable/layout.html#configuring-indent-locations
indent_unit = space
tab_space_size = 4
indented_joins = False
indented_ctes = False
indented_using_on = True
indented_on_contents = True
indented_then = True
indented_then_contents = True
allow_implicit_indents = False
template_blocks_indent = True
# This is a comma seperated list of elements to skip
# indentation edits to.
skip_indentation_in = script_content
# If comments are found at the end of long lines, we default to moving
# them to the line _before_ their current location as the convention is
# that a comment precedes the line it describes. However if you prefer
# comments moved _after_, this configuration setting can be set to "after".
trailing_comments = before
# To exclude comment lines from indentation entirely set this to "True".
ignore_comment_lines = False

# Layout configuration
# See https://docs.sqlfluff.com/en/stable/layout.html#configuring-layout-and-spacing
[sqlfluff:layout:type:comma]
spacing_before = touch
line_position = trailing

[sqlfluff:layout:type:binary_operator]
spacing_within = touch
line_position = leading

[sqlfluff:layout:type:statement_terminator]
spacing_before = touch
line_position = trailing

[sqlfluff:layout:type:end_of_file]
spacing_before = touch

[sqlfluff:layout:type:set_operator]
line_position = alone:strict

[sqlfluff:layout:type:start_bracket]
spacing_after = touch

[sqlfluff:layout:type:end_bracket]
spacing_before = touch

[sqlfluff:layout:type:start_square_bracket]
spacing_after = touch

[sqlfluff:layout:type:end_square_bracket]
spacing_before = touch

[sqlfluff:layout:type:start_angle_bracket]
spacing_after = touch

[sqlfluff:layout:type:end_angle_bracket]
spacing_before = touch

[sqlfluff:layout:type:casting_operator]
spacing_before = touch
spacing_after = touch:inline

[sqlfluff:layout:type:slice]
spacing_before = touch
spacing_after = touch

[sqlfluff:layout:type:dot]
spacing_before = touch
spacing_after = touch

[sqlfluff:layout:type:comparison_operator]
spacing_within = touch
line_position = leading

[sqlfluff:layout:type:assignment_operator]
spacing_within = touch
line_position = leading

[sqlfluff:layout:type:object_reference]
spacing_within = touch:inline

[sqlfluff:layout:type:numeric_literal]
spacing_within = touch:inline

[sqlfluff:layout:type:sign_indicator]
spacing_after = touch:inline

[sqlfluff:layout:type:tilde]
spacing_after = touch:inline

[sqlfluff:layout:type:function_name]
spacing_within = touch:inline
spacing_after = touch:inline

[sqlfluff:layout:type:array_type]
spacing_within = touch:inline

[sqlfluff:layout:type:typed_array_literal]
spacing_within = touch

[sqlfluff:layout:type:sized_array_type]
spacing_within = touch

[sqlfluff:layout:type:struct_type]
spacing_within = touch:inline

[sqlfluff:layout:type:bracketed_arguments]
spacing_before = touch:inline

[sqlfluff:layout:type:typed_struct_literal]
spacing_within = touch

[sqlfluff:layout:type:semi_structured_expression]
spacing_within = touch:inline
spacing_before = touch:inline

[sqlfluff:layout:type:array_accessor]
spacing_before = touch:inline

[sqlfluff:layout:type:colon]
spacing_before = touch

[sqlfluff:layout:type:colon_delimiter]
spacing_before = touch
spacing_after = touch

[sqlfluff:layout:type:path_segment]
spacing_within = touch

[sqlfluff:layout:type:sql_conf_option]
spacing_within = touch

[sqlfluff:layout:type:sqlcmd_operator]
# NOTE: This is the spacing between the operator and the colon
spacing_before = touch

[sqlfluff:layout:type:comment]
spacing_before = any
spacing_after = any

[sqlfluff:layout:type:pattern_expression]
# Snowflake pattern expressions shouldn't have their spacing changed.
spacing_within = any

[sqlfluff:layout:type:placeholder]
# Placeholders exist "outside" the rendered SQL syntax
# so we shouldn't enforce any particular spacing around
# them.
spacing_before = any
spacing_after = any

[sqlfluff:layout:type:common_table_expression]
# The definition part of a CTE should fit on one line where possible.
# For users which regularly define column names in their CTEs they
# may which to relax this config to just `single`.
spacing_within = single:inline

# By setting a selection of clauses to "alone", we hint to the reflow
# algorithm that in the case of a long single line statement, the
# first place to add newlines would be around these clauses.
# Setting this to "alone:strict" would always _force_ line breaks
# around them even if the line isn't too long.
[sqlfluff:layout:type:select_clause]
line_position = alone

[sqlfluff:layout:type:where_clause]
line_position = alone

[sqlfluff:layout:type:from_clause]
line_position = alone

[sqlfluff:layout:type:join_clause]
line_position = alone

[sqlfluff:layout:type:groupby_clause]
line_position = alone

[sqlfluff:layout:type:orderby_clause]
# NOTE: Order by clauses appear in many places other than in a select
# clause. To avoid unexpected behaviour we use `leading` in this
# case rather than `alone`.
line_position = leading

[sqlfluff:layout:type:having_clause]
line_position = alone

[sqlfluff:layout:type:limit_clause]
line_position = alone

# Template loop tokens shouldn't dictate spacing around them.
[sqlfluff:layout:type:template_loop]
spacing_before = any
spacing_after = any

[sqlfluff:templater]
unwrap_wrapped_queries = True

[sqlfluff:templater:jinja]
apply_dbt_builtins = True

# Some rules can be configured directly from the config common to other rules
[sqlfluff:rules]
allow_scalar = True
single_table_references = consistent
unquoted_identifiers_policy = all

[sqlfluff:rules:capitalisation.keywords]
# Keywords
capitalisation_policy = consistent
# Comma separated list of words to ignore for this rule
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:capitalisation.identifiers]
# Unquoted identifiers
extended_capitalisation_policy = consistent
# Comma separated list of words to ignore for this rule
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:capitalisation.functions]
# Function names
extended_capitalisation_policy = consistent
# Comma separated list of words to ignore for this rule
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:capitalisation.literals]
# Null & Boolean Literals
capitalisation_policy = consistent
# Comma separated list of words to ignore for this rule
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:capitalisation.types]
# Data Types
extended_capitalisation_policy = consistent
# Comma separated list of words to ignore for this rule
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:ambiguous.join]
# Fully qualify JOIN clause
fully_qualify_join_types = inner

[sqlfluff:rules:ambiguous.column_references]
# GROUP BY/ORDER BY column references
group_by_and_order_by_style = consistent

[sqlfluff:rules:aliasing.table]
# Aliasing preference for tables
aliasing = explicit

[sqlfluff:rules:aliasing.column]
# Aliasing preference for columns
aliasing = explicit

[sqlfluff:rules:aliasing.length]
min_alias_length = None
max_alias_length = None

[sqlfluff:rules:aliasing.forbid]
# Avoid table aliases in from clauses and join conditions.
# Disabled by default for all dialects unless explicitly enabled.
# We suggest instead using aliasing.length (AL06) in most cases.
force_enable = False

[sqlfluff:rules:convention.not_equal]
# Consistent usage of preferred "not equal to" comparison
preferred_not_equal_style = consistent

[sqlfluff:rules:convention.select_trailing_comma]
# Trailing commas
select_clause_trailing_comma = forbid

[sqlfluff:rules:convention.count_rows]
# Consistent syntax to count all rows
prefer_count_1 = False
prefer_count_0 = False

[sqlfluff:rules:convention.terminator]
# Semi-colon formatting approach
multiline_newline = False
require_final_semicolon = False

[sqlfluff:rules:convention.blocked_words]
# Comma separated list of blocked words that should not be used
blocked_words = None
blocked_regex = None
match_source = False

[sqlfluff:rules:convention.quoted_literals]
# Consistent usage of preferred quotes for quoted literals
preferred_quoted_literal_style = consistent
# Disabled for dialects that do not support single and double quotes for quoted literals (e.g. Postgres)
force_enable = False

[sqlfluff:rules:convention.casting_style]
# SQL type casting
preferred_type_casting_style = consistent

[sqlfluff:rules:references.from]
# References must be in FROM clause
# Disabled for some dialects (e.g. bigquery)
force_enable = False

[sqlfluff:rules:references.qualification]
# Comma separated list of words to ignore for this rule
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:references.consistent]
# References must be consistently used
# Disabled for some dialects (e.g. bigquery)
force_enable = False

[sqlfluff:rules:references.keywords]
# Keywords should not be used as identifiers.
unquoted_identifiers_policy = aliases
quoted_identifiers_policy = none
# Comma separated list of words to ignore for this rule
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:references.special_chars]
# Special characters in identifiers
unquoted_identifiers_policy = all
quoted_identifiers_policy = all
allow_space_in_identifier = False
additional_allowed_characters = None
ignore_words = None
ignore_words_regex = None

[sqlfluff:rules:references.quoting]
# Policy on quoted and unquoted identifiers
prefer_quoted_identifiers = False
prefer_quoted_keywords = False
ignore_words = None
ignore_words_regex = None
force_enable = False

[sqlfluff:rules:layout.long_lines]
# Line length
ignore_comment_lines = False
ignore_comment_clauses = False

[sqlfluff:rules:layout.select_targets]
wildcard_policy = single

[sqlfluff:rules:structure.subquery]
# By default, allow subqueries in from clauses, but not join clauses
forbid_subquery_in = join

[sqlfluff:rules:structure.join_condition_order]
preferred_first_table_in_join_clause = earlier