从文本字符串创建关键字 - 为西里尔文编码问题

I was searching for a code that can automatically create keywords from text string. The code below works fine with latin characters but if I try to use it with russian(cyrillic) - the result is just commas with white spaces in between.

我正在搜索可以自动从文本字符串创建关键字的代码。下面的代码适用于拉丁字符，但如果我尝试使用俄语（西里尔文） - 结果只是逗号，中间有空格。

I am not sure what the problem is, but I believe it has to do with the encoding. I have tried to encode the string with this

我不确定问题是什么，但我认为它与编码有关。我试图用这个编码字符串

$string = preg_replace('/\s\s+/i', '', mb_strtolower(utf8_encode($string)));

but no luck. The result is the same.

但没有运气。结果是一样的。

Also, I included only russian words in the $stopwords var

另外，我在$ stopwords var中仅包含俄语单词

$stopWords = array('у','ни','на','да','и','нет','были','уже','лет','в','что','их','до','сих','пор','из','но','бы','чтобы','вы','была','было','мы','к','от','с','так','как','не','есть','ее','нам','для','вас','о','них','без');

Here is the original code

这是原始代码

function extractCommonWords($string){
  $stopWords = array('i','a','about','an','and','are','as','at','be','by','com','de','en','for','from','how','in','is','it','la','of','on','or','that','the','this','to','was','what','when','where','who','will','with','und','the','www');

  $string = preg_replace('/\s\s+/i', '', $string); // replace whitespace
  $string = trim($string); // trim the string
  $string = preg_replace('/[^a-zA-Z0-9 -]/', '', $string); // only take alphanumerical characters, but keep the spaces and dashes too…
  $string = strtolower($string); // make it lowercase

  preg_match_all('/\b.*?\b/i', $string, $matchWords);
  $matchWords = $matchWords[0];

  foreach ( $matchWords as $key=>$item ) {
      if ( $item == '' || in_array(strtolower($item), $stopWords) || strlen($item) <= 3 ) {
          unset($matchWords[$key]);
      }
  }   
  $wordCountArr = array();
  if ( is_array($matchWords) ) {
      foreach ( $matchWords as $key => $val ) {
          $val = strtolower($val);
          if ( isset($wordCountArr[$val]) ) {
              $wordCountArr[$val]++;
          } else {
              $wordCountArr[$val] = 1;
          }
      }
  }
  arsort($wordCountArr);
  $wordCountArr = array_slice($wordCountArr, 0, 10);
  return $wordCountArr;}

Any help would be appreciated.

任何帮助，将不胜感激。

1 个解决方案

#1

Cyrillic characters are matched by the following regex and removed from the string:

西里尔字符由以下正则表达式匹配，并从字符串中删除：

$string = preg_replace('/[^a-zA-Z0-9 -]/', '', $string);

To avoid it, match word characters with \w. But you need to use the u (PCRE_UTF8) modifier for \w and friends to match letters from other scripts (including Cyrillic).

要避免它，请将字符与\ w匹配。但你需要使用u（PCRE_UTF8）修饰符来\ w和朋友匹配来自其他脚本（包括西里尔语）的字母。

$string = preg_replace('/[^\w -]+/u', '', $string);

One more thing, the regex /\b.*?\b/i is wrong for several reasons. But if you think about it, it could match the same boundary twice.

还有一件事，正则表达式/\b.*?hb/i出于几个原因是错误的。但是如果你考虑一下，它可以匹配相同的边界两次。

Anyway, I believe you don't need to remove unwanted characters form your string. You're only trying to extract words. You could simply use 1 regex to match them.

无论如何，我相信你不需要从你的字符串中删除不需要的字符。你只想提取单词。你可以简单地使用1个正则表达式来匹配它们。

Regex to match words (with Unicode Scripts, including Cyrillic)

正则表达式匹配单词（使用Unicode脚本，包括西里尔语）

preg_match_all('/\w+/u', $string, $matchWords);

Code

码

function extractCommonWords($string){
    $stopWords = array('i','a','about','an','and','are','as','at','be','by','com','de','en','for','from','how','in','is','it','la','of','on','or','that','the','this','to','was','what','when','where','who','will','with','und','the','www');

    $string = strtolower($string);

    preg_match_all('/\w+/u', $string, $matchWords);
    $matchWords = $matchWords[0];

    foreach ( $matchWords as $key=>$item ) {
      if ( $item == '' || in_array(strtolower($item), $stopWords) || strlen($item) <= 3 ) {
          unset($matchWords[$key]);
      }
    }   
    $wordCountArr = array();
    if ( is_array($matchWords) ) {
      foreach ( $matchWords as $key => $val ) {
          $val = mb_strtolower($val);
          if ( isset($wordCountArr[$val]) ) {
              $wordCountArr[$val]++;
          } else {
              $wordCountArr[$val] = 1;
          }
      }
    }

    arsort($wordCountArr);
    $wordCountArr = array_slice($wordCountArr, 0, 10);
    return $wordCountArr;
}


//for testing purposes
$sc = 'Hello there нам для!';
$result = extractCommonWords($sc);
print_r($result);

^{*Disclaimer: I didn't even read the rest of the code. I'm simply pointing out what was wrong}

*免责声明：我甚至没有阅读其余的代码。我只是指出了什么是错的

rextester demo

rextester演示

#1