Skip to content

Shrinking doesn't seem to work as advertised for one_of/frequency/tree #97

@obrok

Description

@obrok

I'm using stream_data to generate a rather large data structure representing an SQL query. I was having problems shrinking the generated examples - it seemed like only minor details like integer values were being shrunk while the query structure as a whole remained the same. I managed to reduce it to the following property (should work after pasting into stream_data tests):

    property "shrinks towards earlier generators" do
      check all size <- positive_integer(),
                seed <- {integer(), integer(), integer()} do
        {:error, result} =
          one_of([:foo, {:bar, integer()}])
          |> check_all(
            [max_shrinking_steps: 100, initial_size: size, initial_seed: seed],
            fn example -> {:error, example} end
          )

        assert result.nodes_visited < 100
        assert :foo = result.shrunk_failure
      end
    end

Error:

  1) property one_of/1 shrinks towards earlier generators (StreamDataTest)
     test/stream_data_test.exs:199
     Failed with generated values (after 5 successful runs):

         * Clause:    size <- positive_integer()
           Generated: 4

         * Clause:    seed <- {integer(), integer(), integer()}
           Generated: {3, 0, 0}

     match (=) failed
     code:  assert :foo = result.shrunk_failure()
     right: {:bar, 0}

Curiously - it does seem to be dependent on the size, even though integer/1 which is used by one_of seems to operate independently of size. If I just set size <- constant(1), then the property passes.

I did some tracing and it does seem like for small initial_size the initial failure found is {:bar, _} and after some shrinking it gets reduced to :foo but that doesn't seem to happen for a larger initial_size. It doesn't seem to happen when the generator tested is just one_of([:foo, :bar]).

This also affects frequency/1 as it has almost the same implementation with the minor difference of how the option is chosen from the list and at the very least tree/2, which is how I originally found it. I assume it also affects anything else that uses frequency/1.

I'd be happy to make some more attempts at fixing this, but I haven't yet understood the code dealing with shrinking fully, so I wanted to make sure I'm on the right track here and also ask for any advice you might have.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions