Add logical operators to issue and other search - and, not, or

Description

Issue searching is good but simplistic. We need the ability to create more complex searches using logical operators to narrow down results. Right now, multiple search criteria can be entered but every one added is ASSUMED to be an 'AND' logical operator.

Proposal

Provide some simple extensions to search queries to allow for more complex searches

start by allowing logical operators like AND, OR, NOT for issue searches may want to include parenthetical evaluation Review JQL to see for any additional inspiration - Jira Query Language - see links below

Links / references

Reference: JQL - Jira query language - https://confluence.atlassian.com/jirasoftwarecloud/advanced-searching-764478330.html https://blogs.atlassian.com/2013/01/jql-the-most-flexible-way-to-search-jira-14/?_ga=1.2418099.266044726.1485301648 http://blogs.atlassian.com/2013/02/jql-the-most-flexible-way-to-search-jira-24/

@ross.campbell This is in the plan. It's mentioned in the original issue for the new search bar:

• https://gitlab.com/gitlab-org/gitlab-ce/issues/21747

@victorwu I'm not sure there's an open issue now that details the plans for the next iteration of the search bar. Where should I be pointing to?

added frontend feature proposal ~14281 labels

@markglenfletcher : https://gitlab.com/gitlab-org/gitlab-ce/issues/25913

mentioned in issue #25913 (moved)

@victorwu Thanks.

@ross.campbell I've edited the issue description for the issue mentioned above (https://gitlab.com/gitlab-org/gitlab-ce/issues/25913) to refer to this issue with regards to "boolean operators"

Highly required by users who take your issue board seriously. I'd consider to add this to Enterprise Starter Edition.

mentioned in issue #13650 (moved)

added ~897282 label

changed milestone to %Backlog

added ~19492 label

@cperessini @awhildy : Would it be possible to do some UX here for a potential 9.1 release? How about we get some UX concepts first, and then carve out the smallest piece? I can see this getting pretty complicated with the different math rules, parentheses, etc. So much so that JIRA invented JQL. I'm think we agree we want to avoid that. Things I'm thinking about are:

• Is everything keyboard-based only? Or do we allow mouse interactions/selections also?
• Complicated expressions. Do we really need them?
• How to handle precedence, symbols that handle those (i.e. parentheses and brackets, etc.)

mentioned in issue #27293 (closed)

@smcgivern : Is this one going to be a major BE one? How about if we scope it down real small, like just supporting "not UX"

@victorwu yes, it will be major, even if we just support one non-AND operation. That's because we just use query params at the moment, which are fine for AND-ing everything together, but we need to:

1. Come up with a way to represent whatever new operation we're adding.
2. That will also be reasonably future-proof to the things that are coming down the line.
3. And be backwards-compatible with our existing scheme, so we don't break current URLs.
4. Without breaking when it's combined with the existing scheme

This is without even going into the code-level details of how we support this efficiently!

@awhildy @jschatz1 @ClemMakesApps : Given the BE complexity that @smcgivern explained above, I'm leaning toward bumping this lower in priority. And instead focus more on breadth, i.e. moving the search bar to other places of GitLab and supporting the light FE functions, like saving searches in browser localStorage.

Have you folks seen a lot of requests for this generic logical search input? My impression is that there are more individual use cases, such as searching for not-WIP. But is anybody asking for this powerful generic logical search?

@awhildy @cperessini : Sorry I'll be sure to ask engineers first next time. Let's not worry about the design here just yet, until we get a better handle of when we will schedule it and the engineering involved.

Yeah, the logical operators will be quite difficult on the BE side even from my limited understanding/perspective. But I do think it would be very useful for power users and for more advanced reporting.

With that being said, there are lot of other things we can do in the meantime

The logical operations will be quite useful, and are in part the sell of this new model. While I totally understand the complexity, and get addressing some of the simpler issues first, don't want this to get pushed down too far the priority I haven't seen a ton of requests, but I do think there are several problems today that people are trying to solve in other ways that would be much simpler with this. Thanks @victorwu!

@victorwu @smcgivern Should we do some work to define the operations we want to support (and the ones we don't) so Backend can have a better idea of how much will need to change?

@cperessini I assumed we were looking to add AND (explicitly; it's implicit at the moment), NOT, and OR, along with the grouping needed to make those work?

@smcgivern that's my idea, too. But there's been talk of doing a simpler version to get started, or to implement NOT, AND and OR now, and then expand on that in the future.

So I thought it may be interesting to specify what the long-term plan is so the implementation can take future versions into account

@cperessini sure thing, that makes sense to me!

@cperessini : It's unlikely we will have this for 9.1. There's other things we can do for the search bar. So while this is important, we can still move forward on the search bar itself in other areas and continue to solve problems and alleviate pain.

As for which specific operators, yes, our long term goal of course we would want to support more complex queries. But we did have one specific issue that requested the NOT condition: https://gitlab.com/gitlab-org/gitlab-ce/issues/13650. So that can be a good place to start only if @smcgivern you want to do any technical analysis at all. But again, let's not over-plan or over-analyze until we get there. It's not worth it spec out any designs now.

@smcgivern : I had assumed we would get AND for free since it's implicit now. I guess I am wrong.

@victorwu you're pretty much right, we have an implicit AND now. I was just assuming that we'd need to specify a precedence for it, and potentially allow grouping, so (label:X AND milestone:M) OR (label:Y AND milestone:N).

mentioned in issue #3012 (closed)

for the meantime a userscript for blurring or hiding issues by Label: https://github.com/DaniilBabanin/hideIssuesByLable

mentioned in issue #29100 (closed)

mentioned in issue #29362 (closed)

assigned to @cperessini

@cperessini I think you have designs for this already. Once the scope is determined, let's make sure the designs are still in line with the proposal and get this updated since the issue is tagged with ~"coming soon" and UX

mentioned in issue #29786 (closed)

mentioned in issue #30912 (closed)

Just a suggestion from some random bloke from the internets, but could this be split in two stages? A NOT operator would be incredibly useful already by itself and could be a testbed for how the full feature is implemented all the while being much simpler to implement. Of course it would have to fit into a bigger picture and requires a bit of forethought.

added customer label

added SP2 label

ZD: https://gitlab.zendesk.com/agent/tickets/74591

A customer is looking for a solution to this problem:

Given the following issues: 
issue_1: labels=[label_1, label_2] 
issue_2: labels=[label_2, label_3]

When I filter the issues by label_2 and label_1 I expect to see issues that have at least one of the labels, not both. 
Currently, if I filter the issues by l label_2 and label_1 i would only get issue_1 returned. However, I would like to see both issue_1 AND issue_2 as they both have the label_2 in common.

To echo @lloeki's comment, please don't try boil the ocean and roll out the common, simple searches first. Look at what other search engines are doing (e.g. https://www.google.com/advanced_search) and follow their conventions (to make it easy on users).

Personally (and I am a geek who's very comfortable with Boolean algebra), I very much prefer Google's approach to searching to Jira's JQL.

Specifically, initially I would NOT worry about nesting sub-expressions (parentheses). Sure, sometime it's extremely useful to be able to target things in a fine grained way, but I'd wager to say it's < 2% of how search is used. Having said that, it's really not all that difficult to deal with complex expressions when using parser generators. I've had good success with PEG.js, but Canopy generates both JavaScript and Ruby.

For my company, the most important aspect of this issue is being able to apply an exclusion filter. This is needed for us to be able to do effective backlog grooming, and is holding us back at the moment.

AND is already handled, so if we could simply have a way of doing NOT (maybe take a queue from Google and just do -), we would go a long way. Just like @jongc, we need NOT for grooming.

On the Elasticsearch / EE side of things, we've got https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1770 which will hopefully go in for 9.2.

What I don't know at the moment is how it will interact with filtered issue / MR search... hmm.

OK, currently there's no integration between the two. The filtered search input uses GET .../issues, etc.

They should unify on syntax, if nothing else, but I wonder how difficult it would be to get filtered search and "normal" search using the same backend.

@victorwu so the problem we have here is that we have two things:

• The issues and MR indices have the nice search bar, but are really better described as 'filters' rather than 'searches'. This allows filtering by label, searching for text, filtering by assignee, etc.
• We have a search box in the top right. This can use ES in EE, and as a result can benefit from ES query syntax. https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1770 implements this for ES in EE, but that only applies to the search box in the top right, not to the index pages.

Thanks @smcgivern @nick.thomas . So would this be the target future state?

1. All search functionality runs through a common BE search framework.
2. If ES is configured on, the BE search framework can support additional functionality, such as the functionality described in this issue.
3. The top right search boxes in all pages of GitLab get piped into the common BE search framework.
4. The issues/board/mr search bar searches get piped into the common BE search framework.
5. In CE, we do not provide the functionality described in this issue, since it requires ES.
6. In EES, we do provide the functionality described in this issue.

Let's agree on the target future state and then plan for steps toward it. I don't think it makes sense to introduce these logical search features here if it will add to more tech debt when we have a chance now to correct it.

Aw, this would mean even though we have EE here we will have to set up ES "just" to get ´NOT´ queries? Currently we don't have ES as we aim to keep operational dependencies to a minimum so as to reduce the maintenance surface. Is there any plan to have ES in the Omnibus package? Also I understand you're aiming for the best design but "unified search framework" sounds like quite a long shot. I don't know what other people think of this but around here ´OR´ is icing on the cake while ´NOT´ is direly needed (worked around by creating labels such as ´No priority´ and trying our best to ensure no issue is unlabeled, which is a pain of epic scale).

Agree with @lloeki and others on this issue. 'OR' or grouped expressions are the vast minority of the use cases for filtering issues. They could be useful and surely a nice to have feature, but I don't think anyone would consider them a priority, especially given their complexity.

Searching for issues without certain labels is instead very common and widespread, and it is crucial to ensure that no issue gets lost in the backlog without proper tagging. Imagine for example having to find issues have not been assigned an area (backend/frontend/UX), or open issues that are neither 'to do' or 'doing' (eg, missing from the board). It can get easily overwhelming with the current feature set, even for organizations <100 developers.

@victorwu yes, but that's quite far away. I think supporting NOT would be something we could look at if we want to do something in the interim for the filters (my first bullet point above). The problem is that if we want to unify things in the future, it should be forwards-compatible with Elasticsearch.

@victorwu there's a distinction to be made between advanced querying of text content like issue descriptions - which needs elasticsearch - and advanced querying of metadata like labels - which does not.

This issue seems to be more about the latter than the former.

We can and should move to a common search backend, but that can happen independently of adding, e.g., the much-requested NOT label syntax, and without introducing significant technical debt.

For the issues / mr search bar, let's do not the condition first, i.e.: https://gitlab.com/gitlab-org/gitlab-ce/issues/32025

mentioned in issue #30468 (closed)

changed milestone to %Next 2-3 months

It would be often important to exclude some issues by using operators like not

mentioned in issue #19808 (closed)

mentioned in issue #34884 (closed)

mentioned in issue #21747 (closed)

Hi, will this be supported by the API when it's realized or do we need to open an additional issue for that?

Hi, will this be supported by the API when it's realized or do we need to open an additional issue for that?

@sils

It would be better if you open a new issue for that proposal if you're still interested.

mentioned in issue #37207

Whats the plan for this issue? :)

@jdalrymple the milestone indicates we will tackle it in the next 2-3 months. :smile.

Desperate for some sort of Not filter, more complicated than that is handy but not essential.

Another customer request for this: https://gitlab.zendesk.com/agent/tickets/82966.

@jschatz1 @jdalrymple perhaps not...

mentioned in issue csoriano/testGitLab#1 (closed)

mentioned in issue #38639 (closed)

@chrisnicola AND why do you say that? Are you NOT sure? OR is something else? :)

mentioned in issue gitlab#7524

moved to gitlab#8161

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue