Keyword alerts or AI: how to test a Telegram request filter

Why can’t your notifications tell you if a filter works?

Noise is easy to spot: a colleague’s ad, a full-time vacancy. A missed request leaves no trace, so the filter seems to err only one way. Testing rules on chat history has the same blind spot.

You need two numbers. Precision is the share of notifications that are useful to you. Recall is the share of all fitting messages the filter found, and to work it out you have to know how many there were.

We haven’t run a general comparison of keywords and AI: the answer depends on your groups and your wording.

• Google: accuracy, precision and recall

How do you build a test sample?

1. 1

   Same groups, same period

   Copy 50 to 100 consecutive messages from your groups. If requests are rare, take more: with five fitting messages, one message moves recall by 20 points.

2. 2

   Label before you look

   Mark each message as fits, doesn’t fit or unclear before you open any notifications, or you’ll catch yourself agreeing with the filter.

3. 3

   Hold back a third

   Set a third of the sample aside while you tune keywords or the instruction, and count the final result on it, so the rules don’t get fitted to messages you already know.

4. 4

   Run the filters

   Test a keyword rule yourself with a text search over the copied messages. Pingfinch and other tools that read only new messages need a week ahead: at the end of it, label those days, then open the notifications.

The guide on grey areas shows how to tell a request from an ad or a job post while labelling.

• Client request, ad or job post? Grey areas on Telegram

What does the count look like?

Four made-up messages and the rule “the text contains ‘deck’”:

The rule showed three messages and one is useful: precision 1 of 3. Of the two fitting messages it found one: recall 1 of 2.

What shouldn’t count as a filter mistake?

Skip overall accuracy altogether. In the example, a filter that stays silent all week is right about 52 of 60 messages, roughly 87%, and finds no requests at all.

Which matters more: less noise or fewer misses?

Filter A means more reading; filter B loses half the requests. With a few large projects a month, recall matters more. With a steady stream of small jobs and little time, precision does. Decide which costs you more: ten extra notifications or one lost client.

After each change, recount on a fresh week and write the raw numbers and dates next to the percentage: “6 of 8, 14 to 20 October” says more than “75%”. When Pingfinch’s AI is in doubt, it rejects the message: less noise, but some requests may be lost, and your sample will show how many.

How do you test Pingfinch the same way?

Pingfinch, a service that notifies you in Telegram about requests in the public groups you choose, doesn’t select by keywords: an AI model decides from your instruction whether to send each message. Before that, it drops messages without a single word, copies of already rejected ones and messages with words you excluded in the subscription’s “🔇 Don't send” rules. The AI sees one message without the conversation, so “Could you do it by tomorrow evening instead?” won’t reach you: count it as an expected miss.

Each notification shows the subscription name, the reason, the source and a link to the message, so checking it against your labels takes one tap. The AI errs both ways, so count its mistakes as you did for keywords.

Where do you start?

Label messages from your own groups, work out precision and recall for a keyword rule, and connect Pingfinch to the same groups for a week. You can start free in the bot; the terms after the free period are on the Pricing page.

• How to filter client requests in Telegram groups: templates
• Telegram monitoring bots and tools: what to compare
• How Pingfinch works
• Pricing

FAQ

Can I find missed requests without reading every message?

Not fully. You can read the messages the filter didn’t show for one day a week and work out recall for that day only.