IAHLT (Israeli Association for Human Language Technologies)

An association the state set up, not a startup

The Israeli Association for Human Language Technologies is not a commercial company but a registered nonprofit association. The Israel Innovation Authority and the Israel National Digital Agency approved its creation in September 2020, and allocated it NIS 7.5 million in funding for three years. Among the bodies that joined it at the founding stage: Rafael, Ginger Software, Melingo, AudioCodes, Intel, Walla, ynet and Bank Hapoalim. Aviv Zeevi, VP of technology infrastructure at the Innovation Authority, explained at the time that the association was meant to let the industry itself lead in defining its needs and help close technology gaps.

The problem behind it: Hebrew and Arabic get poor service

The official announcement of its founding said the association was created in light of the "low and insufficient quality of recognition of the Hebrew and Arabic languages in various kinds of computer systems, compared with speech recognition in other languages." The explanation given was that both languages are Semitic and therefore harder for computers to analyze. The economic logic behind the structure is just as simple: no single company rushes to fund on its own a basic infrastructure that its competitors will also benefit from, so it is built once and jointly, instead of each one repeating it separately.

The open resources: Hebrew and Arabic treebanks

Some of the organization's outputs are published to the general public for free. The three open datasets are a treebank of contemporary Hebrew with 5,000 sentences sampled from Wikipedia articles in seven content areas, among them biography, events, places, health, law and finance; another treebank with 2,500 sentences from Knesset protocols; and an Arabic morphology dataset with 2,000 sentences. A treebank is a collection of sentences analyzed by hand according to their grammatical structure. Alongside the data itself, scripts and trained models are also released.

The membership model: the large datasets are reserved for members

Alongside the open resources, the organization works on a mixed model: the truly large datasets are available only to bodies that have joined as members and share the burden. They include a Hebrew treebank with 48,000 sentences from varied sources, such as news, Wikipedia, government protocols and public information materials; named entity recognition corpora covering more than 123,000 paragraphs in the two languages; a broader Arabic morphology dataset; and transcripts of spoken Palestinian Arabic totaling 93,000 tokens, taken from video clips.

Coreference and named entities, and why it matters for every AI tool in Hebrew

One of the newer open datasets deals with coreference: linking different mentions in the same text to the same entity, for example a first name, a pronoun and a title that all refer to one person. The dataset includes 910 documents, 28,141 clusters and 121,067 annotated mentions. The organization also publishes models for named entity recognition in Hebrew and Arabic on Hugging Face. Materials like these are the raw material used to train and measure models: without them, it is hard even to determine objectively how well a given model understands Hebrew, rather than just producing text that sounds plausible.