How to Dynamically Construct Complex Predicates for SwiftData

NSCompoundPredicate allows developers to combine multiple NSPredicate objects into a single compound predicate. This mechanism is particularly suited for scenarios that require data filtering based on multiple criteria. However, in the new Foundation framework restructured with Swift, the direct functionality corresponding to NSCompoundPredicate is missing. This change poses a significant challenge for developers who wish to build applications using SwiftData. This article aims to explore how to dynamically construct complex predicates that meet the requirements of SwiftData, utilizing PredicateExpression, under the current technical conditions.

Challenge: Implementing Flexible Data Filtering Capabilities

During the development of the new version of the Health Notes app, I decided to replace the traditional Core Data with SwiftData to leverage the modern features of the Swift language. One of the core aspects of this data-centric application is to provide users with flexible and powerful data filtering capabilities. In this process, I faced a key challenge: how to construct diversified filtering schemes for user data retrieval. Here are some predicates used for fetching Memo instances:

extension Memo {
  public static func predicateFor(_ filter: MemoPredicate) -> Predicate<Memo>? {
    var result: Predicate<Memo>?
    switch filter {
    case .filterAllMemo:
      // nil
      break
    case .filterAllGlobalMemo:
      result = #Predicate<Memo> { $0.itemData == nil }
    case let .filterAllMemoOfRootNote(noteID):
      result = #Predicate<Memo> {
        if let itemData = $0.itemData, let item = itemData.item, let note = item.note {
          return note.persistentModelID == noteID || note.parent?.persistentModelID == noteID
        } else {
          return false
        }
      }
    case .filterMemoWithImage:
      result = #Predicate<Memo> { $0.hasImages }
    case .filterMemoWithStar:
      result = #Predicate<Memo> { $0.star }
    case let .filterMemoContainsKeyword(keyword):
      result = #Predicate<Memo> {
        if let content = $0.content {
          return content.localizedStandardContains(keyword)
        } else {
          return false
        }
      }
    }
    return result
  }
}

In the early versions of the application, users could flexibly combine filtering conditions, such as incorporating star marks with images or filtering by specific notes and keywords. Previously, such dynamic combination requirements could be easily achieved using NSCompoundPredicate, which allows developers to dynamically combine multiple predicates and use the result as the retrieval condition for Core Data. However, after switching to SwiftData, I found the corresponding functionality to dynamically combine Swift Predicates was missing, which posed a significant constraint on the core functionality of the application. Addressing this issue is crucial for maintaining the functionality of the app and the satisfaction of its users.

Combining NSPredicate Methods

NSCompoundPredicate offers a powerful way for developers to dynamically combine multiple NSPredicate instances into a single compound predicate. Here is an example demonstrating how to use the AND logical operator to combine two separate predicates a and b into a new predicate:

let a = NSPredicate(format: "name = %@", "fat")
let b = NSPredicate(format: "age < %d", 100)
let result = NSCompoundPredicate(type: .and, subpredicates: [a, b])

Moreover, since NSPredicate allows for construction via strings, developers can utilize this feature to manually build new predicate expressions by combining the predicateFormat property. This method offers additional flexibility, enabling developers to directly manipulate and combine the string representations of existing predicates:

let a = NSPredicate(format: "name = %@", "fat")
let b = NSPredicate(format: "age < %d", 100)
let andFormatString = a.predicateFormat + " AND " + b.predicateFormat // name == "fat" AND age < 100
let result = NSPredicate(format: andFormatString)

Unfortunately, while these methods are very effective and flexible when using NSPredicate, they are not applicable to Swift Predicate. This means that when transitioning to using SwiftData, we need to explore new ways to achieve similar dynamic predicate combination functionality.

The Challenge of Combining Swift Predicates

In the previous article “Swift Predicate: Usage, Composition, and Considerations,” we explored the structure and composition of Swift Predicates in detail. In essence, developers construct the Predicate structure by declaring types that conform to the PredicateExpression protocol. Due to the potentially complex nature of this process, Foundation provides the #Predicate macro to simplify the operation.

When we build Swift Predicates, the #Predicate macro automatically converts these operators into corresponding predicate expressions:

let predicate = #Predicate<People> { $0.name == "fat" && $0.age < 100 }

After the macro is expanded, we can see the detailed composition of the predicate expression:

Foundation.Predicate<People>({
    PredicateExpressions.build_Conjunction(
        lhs: PredicateExpressions.build_Equal(
            lhs: PredicateExpressions.build_KeyPath(
                root: PredicateExpressions.build_Arg($0),
                keyPath: \.name
            ),
            rhs: PredicateExpressions.build_Arg("fat")
        ),
        rhs: PredicateExpressions.build_Comparison(
            lhs: PredicateExpressions.build_KeyPath(
                root: PredicateExpressions.build_Arg($0),
                keyPath: \.age
            ),
            rhs: PredicateExpressions.build_Arg(100),
            op: .lessThan
        )
    )
})

Here, PredicateExpressions.build_Conjunction creates a PredicateExpressions.Conjunction expression corresponding to the && operator. It connects two expressions that return Boolean values, forming a complete expression. Theoretically, if we could extract and combine expressions from Swift Predicates individually, we could dynamically combine predicates based on the AND logic.

The expression types corresponding to || and ! are PredicateExpressions.Disjunction and PredicateExpressions.Negation, respectively.

Given that Swift Predicate provides an expression attribute, it's natural to consider utilizing this attribute to achieve such dynamic combinations:

let a = #Predicate<People1> { $0.name == "fat"}
let b = #Predicate<People1> { $0.age < 10 }
let combineExpression = PredicateExpressions.build_Conjunction(lhs: a.expression, rhs: b.expression)

However, attempting the above code results in a compilation error:

Type 'any StandardPredicateExpression<Bool>' cannot conform to 'PredicateExpression'

A deeper exploration into the implementation of the Predicate structure and PredicateExpressions.Conjunction reveals the constraints involved:

public struct Predicate<each Input> : Sendable {
    public let expression : any StandardPredicateExpression<Bool>
    public let variable: (repeat PredicateExpressions.Variable<each Input>)
    
    public init(_ builder: (repeat PredicateExpressions.Variable<each Input>) -> any StandardPredicateExpression<Bool>) {
        self.variable = (repeat PredicateExpressions.Variable<each Input>())
        self.expression = builder(repeat each variable)
    }
    
    public func evaluate(_ input: repeat each Input) throws -> Bool {
        try expression.evaluate(
            .init(repeat (each variable, each input))
        )
    }
}

extension PredicateExpressions {
    public struct Conjunction<
        LHS : PredicateExpression,
        RHS : PredicateExpression
    > : PredicateExpression
    where
        LHS.Output == Bool,
        RHS.Output == Bool
    {
        public typealias Output = Bool
        
        public let lhs: LHS
        public let rhs: RHS
        
        public init(lhs: LHS, rhs: RHS) {
            self.lhs = lhs
            self.rhs = rhs
        }
        
        public func evaluate(_ bindings: PredicateBindings) throws -> Bool {
            try lhs.evaluate(bindings) && rhs.evaluate(bindings)
        }
    }
    
    public static func build_Conjunction<LHS, RHS>(lhs: LHS, rhs: RHS) -> Conjunction<LHS, RHS> {
        Conjunction(lhs: lhs, rhs: rhs)
    }
}

The issue lies in the expression property being of the type any StandardPredicateExpression<Bool>, which doesn't contain sufficient information to identify the specific PredicateExpression implementation type. Since Conjunction requires the exact types of the left and right sub-expressions for initialization, we are unable to use the expression property directly to dynamically construct new combined expressions.

Dynamic Predicate Construction Strategy

Although we cannot directly utilize the expression attribute of Swift Predicate, there are still alternative ways to achieve the goal of dynamically constructing predicates. The key lies in understanding how to extract or independently create expressions from existing predicates and utilize expression builders such as build_Conjunction or build_Disjunction to generate new predicate expressions.

Utilizing the #Predicate Macro To Construct Expressions

Directly constructing predicates based on expression types can be quite cumbersome. A more practical method is to use the #Predicate macro, allowing developers to indirectly build and extract predicate expressions. This approach is inspired by the contribution of community member nOk on Stack Overflow.

For example, consider the predicate built using the #Predicate macro:

let filterByName = #Predicate<People> { $0.name == "fat" }

By examining the code expanded from the macro, we can extract the part of the code that forms the predicate expression.

Since constructing an instance of PredicateExpression requires different parameters based on the type of expression, the following method cannot be used to generate the correct expression:

let expression = PredicateExpressions.build_Equal(
  lhs: PredicateExpressions.build_KeyPath(
      root: PredicateExpressions.build_Arg($0), // error: Anonymous closure argument not contained in a closure
      keyPath: \.name
  ),
  rhs: PredicateExpressions.build_Arg("fat")
)

Although we cannot directly replicate the expression to create a new PredicateExpression instance, we can redefine the same expression using a closure:

let expression = { (people: PredicateExpressions.Variable<People>) in
  PredicateExpressions.build_Equal(
      lhs: PredicateExpressions.build_KeyPath(
          root: PredicateExpressions.build_Arg(people),
          keyPath: \.name
      ),
      rhs: PredicateExpressions.build_Arg("fat")
  )
}

Creating Parameterized Expression Closures

Since the right-hand side value of the expression (such as "fat") might need to be dynamically assigned, we can design a closure that returns another expression closure. This allows the name to be determined at runtime:

let filterByNameExpression = { (name: String) in
  { (people: PredicateExpressions.Variable<People>) in
    PredicateExpressions.build_Equal(
      lhs: PredicateExpressions.build_KeyPath(
        root: PredicateExpressions.build_Arg(people),
        keyPath: \.name
      ),
      rhs: PredicateExpressions.build_Arg(name)
    )
  }
}

Using this closure that returns an expression, we can dynamically construct the predicate:

let name = "fat"
let predicate = Predicate<People>(filterByNameExpression(name))

Combining Expressions To Construct New Predicates

Once we have defined the closures that return expressions, we can use PredicateExpressions.build_Conjunction or other logical constructors to create new predicates containing complex logic:

// #Predicate<People> { $0.age < 10 }
let filterByAgeExpression = { (age: Int) in
  { (people: PredicateExpressions.Variable<People>) in
    PredicateExpressions.build_Comparison(
        lhs: PredicateExpressions.build_KeyPath(
            root: PredicateExpressions.build_Arg(people),
            keyPath: \.age
        ),
        rhs: PredicateExpressions.build_Arg(age),
        op: .lessThan
    )
  }
}

// Combine new Predicate
let predicate = Predicate<People> {
  PredicateExpressions.Conjunction(
    lhs: filterByNameExpression(name)($0),
    rhs: filterByAgeExpression(age)($0)
  )
}

The complete process is as follows:

1. Use the #Predicate macro to construct the initial predicate.
2. Extract the expression from the expanded macro code and create a closure that generates the expression.
3. Combine two expressions into a new one using a Boolean logic expression (such as Conjunction), thereby constructing a new predicate instance.
4. Repeat the above steps if multiple expressions need to be combined.

This method, although requiring some additional steps to manually create and combine expressions, provides a possibility for dynamically constructing complex Swift Predicates.

Dynamic Combination of Expressions

Having mastered the complete process from predicate to expression and back to predicate, I now need to create a method that can dynamically combine expressions and generate predicates according to the requirements of my current project.

Drawing inspiration from an example provided by Jeremy Schonfeld on Swift Forums, we can construct a method to dynamically synthesize predicates for retrieving Memo data, as shown below:

extension Memo {
  static func combinePredicate(_ filters: [MemoPredicate]) -> Predicate<Memo> {
    func buildConjunction(lhs: some StandardPredicateExpression<Bool>, rhs: some StandardPredicateExpression<Bool>) -> any StandardPredicateExpression<Bool> {
      PredicateExpressions.Conjunction(lhs: lhs, rhs: rhs)
    }

    return Predicate<Memo>({ memo in
      var conditions: [any StandardPredicateExpression<Bool>] = []
      for filter in filters {
        switch filter {
        case .filterAllMemo:
          conditions.append(Self.Expressions.allMemos(memo))
        case .filterAllGlobalMemo:
          conditions.append(Self.Expressions.allGlobalMemos(memo))
        case let .filterAllMemoOfRootNote(noteID):
          conditions.append(Self.Expressions.memosOfRootNote(noteID)(memo))
        case .filterMemoWithImage:
          conditions.append(Self.Expressions.memoWithImage(memo))
        case .filterMemoWithStar:
          conditions.append(Self.Expressions.memosWithStar(memo))
        case let .filterMemoContainsKeyword(keyword):
          conditions.append(Self.Expressions.memosContainersKeyword(keyword)(memo))
        }
      }
      guard let first = conditions.first else {
        return PredicateExpressions.Value(true)
      }

      let closure: (any StandardPredicateExpression<Bool>, any StandardPredicateExpression<Bool>) -> any StandardPredicateExpression<Bool> = {
        buildConjunction(lhs: $0, rhs: $1)
      }

      return conditions.dropFirst().reduce(first, closure)
    })
  }
}

Usage example:

let predicate = Memo.combinePredicate([.filterMemoWithImage,.filterMemoContainsKeyword(keyword: "fat")])

In the current implementation, due to Swift's strong type system (each filtering logic corresponds to a specific predicate expression type), constructing a flexible and generic combination mechanism similar to NSCompoundPredicate appears relatively complex. The challenge we face is how to maintain type safety while implementing a sufficiently flexible strategy for combining expressions.

For my application scenario, the primary requirement is to handle combinations of the Conjunction (logical AND) type, which is relatively straightforward. If future requirements extend to include Disjunction (logical OR), we will need to introduce additional logical judgments and identifiers in the combination process to flexibly address different logical combination requirements while maintaining code readability and maintainability. This may require more meticulous design to adapt to the variable combination logic while ensuring not to sacrifice Swift's type safety features.

The complete code can be viewed here.

An Implementation Inapplicable to SwiftData

Noah Kamara showcased a snippet of code in his Gist that provides capabilities similar to NSCompoundPredicate, making the combination of Swift Predicates straightforward and convenient. This method appears to be an intuitive and powerful solution:

let people = People(name: "fat", age: 50)
let filterByName = #Predicate<People> { $0.name == "fat" }
let filterByAge = #Predicate<People> { $0.age < 10 }
let combinedPredicate = [filterByName, filterByAge].conjunction()
try XCTAssertFalse(combinedPredicate.evaluate(people)) // return false

Despite its appeal, we cannot adopt this method in SwiftData. Why does this seemingly perfect solution encounter obstacles in SwiftData?

Noah Kamara introduced a custom type named VariableWrappingExpression in the code, which implements the StandardPredicateExpression protocol to encapsulate the expression extracted from the Predicate's expression attribute. This encapsulation method does not involve the specific type of the expression; it merely calls the evaluation method of the encapsulated expression during the predicate evaluation.

struct VariableWrappingExpression<T>: StandardPredicateExpression {
    let predicate: Predicate<T>
    let variable: PredicateExpressions.Variable<T>
    
    func evaluate(_ bindings: PredicateBindings) throws -> Bool {
        // resolve the variable
        let value = try variable.evaluate(bindings)
        
        // create bindings for the expression of the predicate
        let innerBindings = bindings.binding(predicate.variable, to: value)
        
        return try predicate.expression.evaluate(innerBindings)
    }
}

Outside the SwiftData environment, this dynamically combined predicate can function correctly because it directly relies on the evaluation logic of Swift Predicate. However, SwiftData operates differently. When filtering data with SwiftData, it does not directly invoke the evaluation method of Swift Predicate. Instead, SwiftData parses the expression tree in the Predicate's expression attribute and converts these expressions into SQL statements to perform data retrieval in the SQLite database. This means the evaluation process is accomplished by generating and executing SQL commands, operating entirely at the database level.

Therefore, when SwiftData attempts to convert this dynamically combined predicate into SQL commands, the inability to recognize the custom VariableWrappingExpression type results in a runtime error of unSupport Predicate.

If your scenario does not involve using predicates in SwiftData, Noah Kamara's solution might be a good choice. However, if your requirement is to build dynamically combined predicates within the SwiftData environment, you might still need to rely on the strategy introduced in this article.

Optimizing Swift Predicate Expression Compilation Efficiency

Constructing complex Swift Predicate expressions can significantly impact compilation efficiency. The Swift compiler needs to parse and generate complex type information when processing these expressions. When the expressions are overly complex, the time required for the compiler to perform type inference can dramatically increase, slowing down the compilation process.

Consider the following predicate example:

let result = #Predicate<Memo> {
  if let itemData = $0.itemData, let item = itemData.item, let note = item.note {
    return note.persistentModelID == noteID || note.parent?.persistentModelID == noteID
  } else {
    return false
  }
}

In this example, even minor code changes can cause the compilation time for this file to exceed 10 seconds. This delay can also occur when generating expressions using closures. To alleviate this issue, we can utilize Xcode's auxiliary features to clarify the type of the expression. Using Option + Click on the closure reveals the exact type of the expression, allowing us to provide a precise type annotation for the closure's return value.

let memosWithStar = { (memo: PredicateExpressions.Variable<Memo>) -> PredicateExpressions.KeyPath<PredicateExpressions.Variable<Memo>, Bool> in
  PredicateExpressions.build_KeyPath(
    root: PredicateExpressions.build_Arg(memo),
    keyPath: \.star
  )
}

The specific type of the expression in the above complex predicate is shown as follows:

Specifying the expression's type can help the compiler process the code faster, significantly improving the overall compilation efficiency because it avoids the time the compiler spends on type inference.

This strategy is applicable not only in cases where predicates need to be combined but also in situations involving complex predicates without combination. By extracting expressions and specifying types explicitly, developers can significantly improve the compilation time for complex predicates, ensuring a more efficient development experience.

Conclusion

This article explored methods for dynamically constructing complex predicates within the SwiftData environment. Although the current solutions may not be as elegant and straightforward as we might hope, they do provide a viable way for applications relying on SwiftData to implement flexible data querying capabilities without being limited by the absence of certain features.

Despite having found methods that work within the current technological constraints, we still hope that future versions of Foundation and SwiftData will offer built-in support to make constructing dynamic, complex predicates simpler and more intuitive. Enhancing these capabilities would further augment the practicality of Swift Predicate and SwiftData, enabling developers to implement complex data processing logic more efficiently.